These are the steps to configure hprof profiler in GlassFish 3.x:

1, Identify the target JVM to profile.  In most cases, it's the domain administration (DAS) JVM, but it can be other JVM such as another standalone server instance, or a cluster server instance.

2, Edit $GLASSFISH_HOME/config/osgi.properties, locate org.osgi.framework.bootdelegation property, and append hprof classes to it, using , (comma) as the package separator.  Do not forget to add a comma at the end of the existing value.  The resulting property should look like this:

org.osgi.framework.bootdelegation=${eclipselink.bootdelegation}, \
                                  com.sun.btrace, com.sun.btrace.*, \
                                  org.netbeans.lib.profiler, org.netbeans.lib.profiler.*, \
                                  sun.tools.hprof,com.sun.tools.hat.internal.parser,com.sun.demo.jvmti.hprof

3, Start the domain, and go to admin console to add the hprof profiler:

asadmin start-domain
http://localhost:4848

On the left, choose Configurations | server-config | JVM Settings, on the rigth content panel, choose Profiler tab,

Name: hprof
Status: enabled yes
Classpath: not needed
Native library path: no needed
add a JVM Option:

-agentlib:hprof=file=log.txt,thread=y,depth=3

or using the old non-standard option:

-Xrunhprof:file=log.txt,thread=y,depth=3

Create this profiler and restart the domain from the command line.  You will see the following elements are added to $GLASSFISH_HOME/domains/domain1/config/domain.xml,

<profiler name="hprof" native-library-path="" classpath="">
  <jvm-options>-Xrunhprof:file=log.txt,thread=y,depth=3</jvm-options>
</profiler>

So you could also directly edit domain.xml (not recommended) to save you some clicks, if you know where to add this new element.  For server-config, which corresponds to DAS, it is usually after the first group of jvm-options elements.

There is also asadmin CLI commands to create and delete profiler configuration:

asadmin create-profiler hprof

asadmin delete-profiler

But create-profiler command doesn't allow you to specify hprof jvm options, so you would still need to come back to domain.xml to add jvm options.  However, it takes a --target param, where you can specify where to apply the profiler configuration.

4, After starting and stopping the domain, the hprof data file is created in $GLASSFISH_HOME/domains/domain1/config/ directory,  which is the ${user.dir} for GlassFish server process.

Another related post: How to Configure hprof in JBoss AS7