These are the steps to configure hprof profiler in JBoss AS 7.x:

1, Identify the target JVM to profile.  In most cases, it's the standalone server JVM, but it can be other JVM such as domain process controller, host controller, or individual server.

2, For standalone server, edit $JBOSS_HOME/bin/standalone.conf, locate JBOSS_MODULES_SYSTEM_PKGS property, and append hprof classes to it, using , (comma) as the package separator.  This property tells JBoss Modules to make hprof classes accessible from any class loaders.

JBOSS_MODULES_SYSTEM_PKGS="org.jboss.byteman,sun.tools.hprof,com.sun.tools.hat.internal.parser,com.sun.demo.jvmti.hprof"

3, Still in standalone.conf file, uncomment JAVA_OPTS property, and append hprof options:

JAVA_OPTS="$JAVA_OPTS -agentlib:hprof=file=log.txt,thread=y,depth=3"

or using the old non-standard -X option:

JAVA_OPTS="$JAVA_OPTS -Xrunhprof:file=log.txt,thread=y,depth=3"

4, Start the standalone server, and the hprof data file (log.txt) will be created in the current directory.

One common error when enabling hprof in JBoss AS 7.x is a series of NoClassDefFoundError, which indicates JBOSS_MODULES_SYSTEM_PKGS property has not been properly configured as in step 2.

Exception in thread "main" java.lang.NoClassDefFoundError: com/sun/demo/jvmti/hprof/Tracker
    at org.jboss.logmanager.LogManager$1.run(LogManager.java:65)
    at org.jboss.logmanager.LogManager$1.run(LogManager.java:52)
    at java.security.AccessController.doPrivileged(Native Method)
    at org.jboss.logmanager.LogManager.<init>(LogManager.java:52)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
    at java.lang.Class.newInstance0(Class.java:355)
    at java.lang.Class.newInstance(Class.java:308)
    at java.util.logging.LogManager$1.run(LogManager.java:168)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.util.logging.LogManager.<clinit>(LogManager.java:157)
    at org.jboss.modules.Main.main(Main.java:278)
Caused by: java.lang.ClassNotFoundException: com.sun.demo.jvmti.hprof.Tracker from [Module "org.jboss.logmanager:main" from local module loader @1a28362]
    at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:190)
    at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:468)
    at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:456)
    at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:423)
    at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:398)
    at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:120)
    ... 14 more
Exception in thread "Thread-2" java.lang.NoClassDefFoundError: Could not initialize class java.util.logging.LogManager
    at java.util.logging.LogManager$Cleaner.run(LogManager.java:211)
Dumping Java heap ... allocation sites ... done.

Another related post: How to Configure hprof in GlassFish 3.x