Skip to content

Archives

The easy way to find JMX metrics in the field using jmxsh

(oh look, a proper blog post!)

JMX is the de-facto standard in the Java and JVM-based world for exposing service metrics, and feeds nicely to tools like Graphite using JMXTrans and others. However, it’s pretty obtuse and over-complex, and it can be hard to figure out what path the JMX metrics will show up under once deployed.

Unfortunately, once a JVM-based service is deployed to EC2, it becomes very difficult to use jconsole to connect to it, due to deficiencies and crappy design in the JMX RMI protocol (I love the way they reinvented the broken parts of IIOP in that respect). Don’t even bother; instead, use jmxsh: https://code.google.com/p/jmxsh/ .

To use this, you need to modify the service process’ command line to include the following JVM args, so that the remote JMX API is exposed:

-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=16660 -Dcom.sun.management.jmxremote.local.only=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false

Change the port number if there is already a process running on that port. Ensure the port isn’t accessible from off-host; in EC2, this should be safe enough to use once that port number is not in the EC2 security group.

Go to https://code.google.com/p/jmxsh/downloads/list and download the latest jmxsh-FOO.jar; e.g. ‘wget https://jmxsh.googlecode.com/files/jmxsh-R5.jar’. Then on the host, as the UID the service is running under, run: ‘java -jar jmxsh-R5.jar -h 127.0.0.1 -p 16660’. You can then hit “Enter” to go into “Browse Mode”, and you’ll get text menus like this:

 ====================================================

  Attribute List:

        1. -r- long        MaxFileDescriptorCount
        2. -r- long        OpenFileDescriptorCount
        3. -r- long        CommittedVirtualMemorySize
        4. -r- long        FreePhysicalMemorySize
        5. -r- long        FreeSwapSpaceSize
        6. -r- long        ProcessCpuTime
        7. -r- long        TotalPhysicalMemorySize
        8. -r- long        TotalSwapSpaceSize
        9. -r- String      Name
       10. -r- int         AvailableProcessors
       11. -r- String      Arch
       12. -r- double      SystemLoadAverage
       13. -r- String      Version

   SERVER: service:jmx:rmi:///jndi/rmi://127.0.0.1:16660/jmxrmi
   DOMAIN: java.lang
   MBEAN:  java.lang:type=OperatingSystem

 ====================================================

Navigate through the MBean tree looking for good Attributes which would make good metrics (5 in the list above, for example). Note the MBean and the Attribute names.