July 24, 2014

Explicit Full GC is enabled for RMI calls

This is rather strange behavior, but has been recently observed that whenever your application is exposed to services via RMI or it consumes any service over RMI, you will have another garbage collection cycle after every one hr. As explained in RMI documentation:

“When it is necessary to ensure that unreachable remote objects are unexported and garbage collected in a timely fashion, the value of this property represents the maximum interval (in milliseconds) that the Java RMI runtime will allow between garbage collections of the local heap. The default value is 3600000 milliseconds (one hour).”

This will overwrite your JVM algorithm, and will explicitly initiate Full GC.
If you don't want to have this GC happening every one hr than you can use following JVM argument -XX:+DisableExplicitGC   to control it.





May 12, 2014

Jboss Performance Improvements



To update the JBoss XML files:
         standalone.xml
     Open the standalone.xml file in a text editor. and do the following:
a.       In the file, locate “urn:jboss:domain:threads:1.1” and under the code line, add the following:

b.        In the file locate connector name="default" and in the code line add the following parameters after enabled="true"
executor="default-thread" max-connections="1000"
The line will look like:
c.        In the file locate connector name="live" and in the code line add the following parameters after enabled="true"
executor="default-thread" max-connections="1000"
The line will look like:
d.        In the file locate connector name="pushlet" and in the code line add the following parameters after enabled="true"
executor="default-thread" max-connections="1000"

April 14, 2014

IIS 7 performance improvements

Running load test with higher load, its been observed that web server (IIS 7 ) performance degrade significantly. Not much clue has been provided by support.

On further analysis, following setting have been identified as booster to IIS 7 performance :

Updating Default Application Pool Settings

To update default application pool settings:

1.     Go to Start > Administrative Tools > Internet Information Services (IIS) Manager.
2.     In the Internet Information Services (IIS) Manager window, browse to Server_Name > Application Pools.
3.     In the Application Pools section, Right-click DefaultAppPool and from the menu select, Advanced Settings.
4.     In the Advanced Settings window, set the following:
        a.     In the Process Model section, set:
                l     Maximum Worker Processes to 20
                2     Ping Enabled set to False (default value is False)
        b.     In the Rapid-Fail Protection section, set:
                Enabled to False (default value is False)
        c.     In the Recycling section, set: 

                Request Limit to 0 (default value is 0)

        Click OK to close the window.


Updating ASP Settings

To update ASP settings:
1.     Go to Start > Administrative Tools > Internet Information Services (IIS) Manager.
2.     In the Internet Information Services (IIS) Manager window, browse to Server_Name.
3.     In the IIS section, right-click ASP and from the menu select, Open Features.
4.     In the properties screen that opens, go to the Limit Properties section and for the Thread Per Processor Limit setting, set the value to 100.


Updating Registry Settings

To update registry settings:
1.     Go to Run and type regedit to launch the Registry Editor.
2.     In the Registry Editor, browse to HKEY_LOCAL_MACHINE > SYSTEM > CurrentControlSet > services > InetInfo > Parameters.
3.     From the parameters list, right-click MaxPoolThread and from the menu select Modify.
4.     In the Edit DWORD (32-bit) Value window, set the Value data to 400.            
5.     In the Registry Editor, browse to HKEY_LOCAL_MACHINE > Software > Microsoft > ASP.NET > 2.0.50727.0.
6.     In the list section, right-click and from the Menu select New > DWORD (32-bit) Value.
7.     Name the new DWORD as MaxConcurrentRequestsPerCPU.
8.     Right-click MaxConcurrentRequestsPerCPU and from the menu select Modify.
9.     In the Edit DWORD (32-bit) Value window, set the Value data to 5000.
       

May 24, 2012

Error -- memory violation : Exception ACCESS_VIOLATION received

Loadrunner error,

One of the very common issue observed during load testing for large number of users on a single load generator.

  • Default value for Multithreading in Run-time Settings (Virtual User Generator -> General -> Miscellaneous), is set to ‘Run Vuser as a process’.This means that each process of mdrv.exe process will execute a set number of thread (defined in C:\Windows\wlrun5.ini (old versions) or C:\Program Files\HP\LoadRunner\config\wlrun7.ini ), default value is 50.

  • If we have enough memory on load generator machine then we can change setting to ‘Run Vuser as a thread’, which means each mdrv.exe will have single thread instead of default 50.This could be useful in cases where we have heavy transaction (in bytes) on each threads, and in most cases resolve memory violation issue.

  • If we don't have large memory then we can reduce the number of threads running under each mdrv.exe process by reducing value of AgentMaxThreadsPerDriver=50 to lower number.This also helps in getting rid of memory violation, by increasing the number of process of mdrv.exe and reducing load on each process.

  • web_set_max_html_param_len(X000); is set too high to capture maximum data using web_reg_save_param function, try to calculate the max limit of strings per page and add around 30% to it.

  • We also see this during NULL pointer exception while dealing with C functions, best solution is that whenever we expect a NULL value place an If else condition to tackle it


  • You will receive this error if you have defined an array in C and the string length is exceeding the max limit of array.

  • You will receive this error if you are not closing file properly while doing file operations using C functions

  • Size of the string to be updated in single cell of vts is reduced to minimum, try to use flags only, and also keep column name as small as possible.Also increase Think time  between different request to VTS. Lots of write processes will generate this error instead of read.

  • Observed this issue on when load generator machines are on VM (Virtual Machines) instead of Physical box. Moving them to physical box has resolved this if all above mentioned issues are resolved. The best symptom for this is that you observe Memory Violation while running load in controller and after few successful iterations with complete load.


There can be lots of reason for this error share your experiences of this issue and your solution to it.

April 11, 2012

Garbage First (G1) garbage collector (GC)

This new collector has been introduced in Java HotSpot VM in JDK1.7

Difference between current CMS and G1

1. G1 uses all CPUs, which reduces time in "stop-the-world" or GC pauses
2. In G1 Young and Old are not physically separated, and resources can move between two as needed.
3. G1 performs heap compaction over time, which eliminates potential fragmentation problems.

Enable G1 using following options:

      -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC

Set the max GC pause time in milliseconds (max time in GC):

      -XX:MaxGCPauseMillis=50

Set the time interval over which GC pauses for totaling up (wait time before GC):

      -XX:GCPauseIntervalMillis=200

Set the size of the young generation (its depreciated in jdk1.7)
       http://hg.openjdk.java.net/jdk7/jdk7/hotspot/rev/a1c410de27e4

      -XX:+G1YoungGenSize=512m

Set the survivor ratio size

      -XX:SurvivorRatio=6

February 16, 2012

Understanding Concurrent Mark Sweep Garbage Collector Logs


SET JVM_OPTION_X=-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -Xloggc:"%INSTALL_DIR%\logs\printgc_name.log"

verbose:gc log file using different switches.

338.075: [GC 338.079: [ParNew
Desired survivor size 49152000 bytes, new threshold 15 (max 15)
- age   1:    3739224 bytes,    3739224 total
- age   2:    1542896 bytes,    5282120 total
- age   3:   27780808 bytes,   33062928 total
: 612848K->47430K(672000K), 0.0259963 secs] 732630K->167112K(2001152K), 0.0299831 secs] [Times: user=0.13 sys=0.00, real=0.03 secs]

  • Time is in milisecond relative to start of application
  • "[GC" represents a stop-the-world Gargbage Collection triggered by the JVM
  • "[ParNew" represent Young Generation collection.
  • It explains that 672000K is the capacity of Young Generation, and after garbage collection its size drop from 612848K to 47430K in 0.0259963 secs
  • Outside [GC it explains stats of total heap usage dropping from 732630K to 167212K (2001152K) after Young Generation collection.
  • This gives a interesting data,
               732630K - 167212K = 565418K is the amount of data garbage collected in the young generation.
                612848K - 47430K - 565418K = 100K is the amount of data moved into old generation.
  • Describes the "aging distribution" of  objects allocated by application in young generation at the end of young Garbage Collection cycle.
  • This Garbage Collection cycle set the new threshold= 15 meaning that objects will age up to 15 before the next GC cycle takes place.
  • "[Times:" explains the time spend in Young GC collection, user time, system time and real time.


11318.945: [Full GC 11318.946: [CMS11319.283: [CMS-concurrent-sweep: 0.388/0.425 secs] [Times: user=0.66 sys=0.00, real=0.43 secs]
 (concurrent mode failure): 1329151K->1329151K(1329152K), 3.2141208 secs] 2001151K->1395840K(2001152K), [CMS Perm : 90303K->90303K(150656K)], 3.2155193 secs] [Times: user=3.22 sys=0.02, real=3.22 secs]
  • Time is in milisecond relative to start of application
  • "[Full GC" represents a stop-the-world Gargbage Collection triggered by the JVM
  • "[CMS" represent Concurrent Mark and Sweep collection.
  • It explains no change has happenned which doing Full GC which took 3.2
  • Overall heap usage of system come down from 2001151k to 1395840K.

Reasons for (concurrent mode failure):

1. Not enough space in Old generation for larger objects, so young generation is not happening because promotion of objects is not guaranteed. Fragmentation could be the issue here, one option is to increase RAM.
  
2. Objects are getting created at faster rate and CMS is not able to clean them at that rate.  Solution to this problem could be, to reduce CMSInitiatingOccupancyFraction and trigger CMS collection early. By default its value is around 66%

February 10, 2012