Journey of upgrading Grails application from version 1.3 to 2.5

Grails, convention over configuration, provides a great deal of coding by convention, powerful features like integrated ORM, expressive DSL, smooth Java integration, various built in plugins and many many more. However, with all of its beauty, its indeed a double edged sword. It comes with its own pain points. And when it comes to an upgrade, its mostly about knowing the unknowns.

As part of technology upgrade, we took up upgrading our Grails application from version 1.3 to version 2.5 This application was developed on Grails version 1.3 about seven years ago and it worked smoothly with no major issues. But now with the need of moving to higher versions of Java, JBoss, etc and also utilizing power of new features given by newer versions of Grails,  we decided to upgrade our Grails application to a higher version.

During this upgrade we faced numerous issues. Most issues were related to Grails upgrade and some were very much specific to our project. Again, some Grails upgrade issue were major and some were easy and quick to fix. In this post I will try to highlight the major ones that I faced during this upgrade process and their fixes or workarounds.

The application that we were going to upgrade was build on Grails version 1.3.3, we thought migrating directly to the latest version of Grails v2.5 will be a big jump, so from development point of view we decided to go step by step. First step was to migrate to Grails version 2.0.0. This migration was without many issues and quick.

Next step was to move on version 2.5’s latest, that is, version 2.5.5 and this opened the door for issues.

Grails version issue

While trying to run the application on Grails version 2.5.5, I started getting below error:

Error:org.codehaus.groovy#groovy-all;2.4.5: configuration not found in org.codehaus.groovy#groovy-all;2.4.5: ‘master’. It was required from org.grails#grails-core;2.5.5 compile

After struggling for a long time and trying different solutions like adding groovy as dependency etc, it was observed that with Grails version 2.5.5, I was not even able to create a new basic application using grails command

grails-createapp

This too gave the same error.

Solution:

 1. Remove Grails version 2.5.5 (if this is already installed)
 2. Download/install Grails version 2.5.4
 3. Run basic command grails-createapp. This runs fine and an app is created.
 4. Remove Grails version 2.5.4
 5. Install Grails version 2.5.5 now
 6. Run command - grails createapp (this starts working now)

Even though I found the workaround for this behavior of Grails version 2.5.5, we decided to go ahead with version 2.5.4 instead, considering it more stable.

 

Validate method issue

My code had a domain class with one of its fields as Enum. In controller, domainObj.validate() always returned false (even though all the values of domain object were as per the constraints mentioned). Validation error was –

grails.validation.ValidationErrors: 1 errors Field error in object ‘Parameter’ on field ‘typeEnum’: rejected value [0]; codes [com.TypeEnum.typeMismatch.error,com.TypeEnum.typeMismatch,com.TypeEnum.typeMismatch.error,com.TypeEnum.typeMismatch,typeMismatch.com.TypeEnum,typeMismatch.pspValueTypeEnum,typeMismatch.com.TypeEnum,typeMismatch]; arguments [typeEnum]; default message [No enum constant com.TypeEnum.0]

The same code worked absolutely fine with Grails version 1.3.3, but domainObj.validate() in this case always returned false and it started blocking some screens to function.

After a lot of analysis,  2 issues with Grails version 2.5.4 were found.

Existing code:

domainObj.properties = params

// code to convert String from params to Enum object and assign it to domainObj
domainObj.enumField = MyEnumClass.getEnumByName(params.enumNameValue)

domainObj.validate()   //This always returns false with Grails v2.5.4

Issue 1: Enum Field on domain object

If a field (like enumField in my example) is an enum then enum “string” value is expected to be passed to domain object (domainObj) and not the enum object. For example, if my enum is

  1.  enum TypeEnum {
        STRING(0, "String"), NUMBER(1, "Number"), TIME(2, "Time")
         ...
       }

then value to be set into domainObj is “STRING” or “NUMBER”, etc

Issue 2: Sequence issue

Identifying this issue was like solving a puzzle in my code 🙂  In controller, I used

domainObj.properties = params

With Grails version 2.5.4, if I assign anything to domainObj after the above line of code is executed, then domainObj.validate() always return false. If we need to assign some value to one of the fields of domainObj then it has to be updated in the params before executing above line of code. Below example will make this clear.

In below code, domainObj field is updated after executing domain.properties = params, so validate() returns false.

 domainObj.properties = params
 domainObj.enumField = "STRING"
 domainObj.validate()  //always returns false with Grails v2.5.4

If domainObj field is updated in params before executing domainObj.properties = params, then validate() returns expected true.

 params.enumField = "STRING"
 domainObj.properties = params
 domainObj.validate()  // returns True as expected

This sequence is a must to get correct value out of valiadate() method. This issue was nailed downed after a lot of trial and error.

 

Stricter Groovy

Grails v2.5.4 has advanced version of Groovy (v2.4) as compared to earlier Grails version 1.3.3’s Groovy. During the upgrade, we had to fix some groovy issues which the earlier version of Groovy had ignored and worked fine. The newer version of Groovy complaints about such issues. Some of these issues where at compilation  level and easy to fix. For example, missing quotes or missing end tags in GSP files, etc. However, some issues banged at runtime. Such issues were easy to fix but not so easy to figure out where the problem exists. Listed below some of such issues:

Resource loading (Opening a GSP file from another GSP)

In a GSP file (connect.gsp) I was trying to open another gsp rename.gsp. This rename.gsp file is in the same location as that of connect.gsp. However, the file is not found and 404-Resource not found error is shown in the pop-up.

connect.gsp (javascript code)

showPopWin("${request.getContextPath()}/user/rename.gsp", 280, 160, returnFunction);

grails-app
     - views
          -user
              -connect.gsp
              -rename.gsp

Solution:

Places where I have to call a gsp page from another gsp, submit to controller action and from controller action use render to go to respective gsp file.

In connect.gsp,

showPopWin("${request.getContextPath()}/user/renameUI", 280, 160, returnFunction);

In UserController add an action renameUI,

def renameUI= { render( view : 'rename', model:[params:params]) }

Scope of loop parameter “it”

Scope of default parameter(“it”) of loops in GSP pages should be within that particular loop, however we faced issues in places where a single gsp page had multiple loops (g:each statements). Parameter “it” in earlier loop was getting referenced in the subsequent loops. To fix this, we used different names for loop parameters instead of using “it” in each loop. Again, this issue was faced in the upgraded version of Grails – 2.5.4.

 

Assessing private methods

This is a silly one. At times I faced NullPointerExceptions inside some method. After debugging the code it was seen that some object in that method was unexpectedly null. The issue was that the method (inside which some objects were null) was mistakenly marked as private and were accessed from other classes. Older version of Groovy accepted this without any issues but the new version of Groovy started complaining at run time.

Apache Shiro

In Grails version 2.5.4, security framework Jsecurity is rebranded as Apache Shiro.
Jsecurity authentication, its apis, etc are no longer supported, instead Apache Shiro security needs to be used.

Almost all the apis of jsecurity that my application was using had corresponding apis of Apache Shiro. For example, org.jsecurity.SecurityUtils got replaced by org.apache.shiro.SecurityUtils. Also, the code change for authentication, getting auth token, etc was as described in Apache Shiro documentions and was smooth.

Only place where I had to struggle a bit while moving to Apache shiro from jsecurity was the default realm used for hashing the password. After making all the changes related to Apache Shiro, application login was not successful saying credentials don’t match. Problem for this issue is that Apache Shiro’s default realm that hashes the password provided in the authentication token is SHA256, while Jsecurity used Sha1Hash. As all the existing passwords were hashed with Sha1Hash, I had to add below lines into the beans definition in resources.groovy file.

beans = {

    credentialMatcher(Sha1CredentialsMatcher) {
        storedCredentialsHexEncoded = true
   }
}

Maven build

Once my application was stable, working with all green testcases, the next task was setting up the maven script. Things worked all fine using Grails commands like clean, run-app, war, etc. But the Continuous Integration setup that we had needed maven script to build the code and create its war. Grails command generate-pom created pom.xml file (referring the BuildConfig.xml file). After making some script changes to fix issues and adding plugin grails-maven-plugin to clean, build and create war, the script failed to locate classes within jars files located under lib folder.

Tried some workarounds to fix this issue but found no luck. Finally we had to decide that CI will make use of Grails commands to build code, run tests and create war instead of maven. 

Similar pom.xml corresponding to Grails version 1.3.3 worked well. 

With this we are ready to hit the production environment. I have tried to mention only the major or frequent issues during the upgrade. Also, there is change in running the server in debug mode from command prompt, added the setup steps in Debugging Grails 2.5 application.

This is my journey of upgrading Grails application. Please feel free to share your experience of any Grails upgrade.

Advertisements

Debugging Grails application after upgrading from version 1.x to 2.x

As part of technology upgrade, I am upgrading my Grails application from version 1.3 to version 2.5. With Grails version 1.3, debugging within any IDE was pretty straight forward. While I was busy fixing issues related to upgrade, I learned that I was not able to debug the application in the same old way in which we only had to start the application in debug mode from IDE and have it stop at our breakpoints. Fixing issues without being able to debug is as bad as hell.

In the newer version of Grails, Forked execution is introduced. With Grails 2.5, command run-app will launch the Grails application in a new JVM (Java Virtual Machine). I am using version 2.5 but this change is introduced from version 2.2.

To enable this fork setting in application written in older versions like Grails 1.3, one has to add below lines to file grails-app/conf/BuildConfig.groovy

grails.project.fork = [
// configure settings for compilation JVM, note that if you alter the Groovy version forked compilation is required
// compile: [maxMemory: 256, minMemory: 64, debug: false, maxPerm: 256, daemon:true],

// configure settings for the test-app JVM, uses the daemon by default
test: [maxMemory: 768, minMemory: 64, debug: false, maxPerm: 256, daemon:true],
// configure settings for the run-app JVM
run: [maxMemory: 768, minMemory: 64, debug: false, maxPerm: 256, forkReserve:false],
// configure settings for the run-war JVM
war: [maxMemory: 768, minMemory: 64, debug: false, maxPerm: 256, forkReserve:false],
// configure settings for the Console UI JVM
console: [maxMemory: 768, minMemory: 64, debug: false, maxPerm: 256]
]

grails.project.fork.run=true

If one is creating a new application with Grails version 2.x then these settings are available by default in BuildConfig.groovy, these can be changed if needed.

After this setting we can start the application forked debug mode using command

grails run-app –debug-fork

In my case, I got an error related to Tomcat plugin.

ClassNotFoundException - org.apache.catalina.startup.Tomcat

To fix this, I had to make Tomcat plugin available during build in grails-app/conf/BuildConfig.groovy

build “:tomcat:7.0.70”

This opens up the debug port 5005. This will not start the application, it will listen for port 5005. Message seen on console will be

Listening for transport dt_socket at address: 5005

In the IDE GGTS, go to “Debug Configurations” and create a new Remote Java application with connection port 5005.

remote

Save this configuration and then start this in “Debug” mode. Once this is started, we can see the server continuing to start which was listening to port 5005 earlier. After server is started, we can browse the application which will stop at all the breakpoints. Finally ! 🙂

 

Finding root cause of OutOfMemory

For a developer (like me I admit) who has mostly worked on developing/coding applications, things appear to be a bit tricky when questioned as “WHY??” a particular situation occurred in production. Grr !!! Is life all that simple to answer this question anytime perfectly?

When it comes to memory leak analysis for the first time, there are multiple questions running through mind like how to do this, where to start from, what to use, etc.

This is my journey of analyzing heap dump for the first time.

In my organization, a support application was crashed with OutOfMemory error. This issue was escalated to QA team and then was routed to software development team. Although the issue was resolved immediately with a restart of tomcat server on which the application was hosted, there was an RCA request burning on the plate of developers.

For the root cause analysis, we (developers) would indeed need some data. As it was an OOO exception, there has to be a memory dump file created (parameter HeapDumpOnOutOfMemoryError ), so I requested the dump file from the respective team.

I started looking out for tools to open and analyze the dump file and came across a plugin of Eclipse – Memory Analyzer. I got the dump file the next day, and it was hprof rile of more than 5 GBs.

I took the below steps to reach the root cause of this issue using memory analyzer:

  1. Tried to open the hprof file in Eclipse (by double clicking the hprof file in Eclipse)As it was a very huge file and the memory allocated to my Eclipse was just around 1GB, Eclipse complained with Heap Error. To resolve this issue, I increased Eclipse memory to around 6GB (in eclipse.ini file) With this change, the Eclipse error got resolved and it started parsing the dump file.
  2. In my case, it took almost 2 hours to completely parse and open the dump file.
  3. After going through the details of memory analyzer below pointers helped me nail down the root cause:

a. Leak Suspects – Eclipse Memory analyzer tool (MAT) showed 2 Leak Suspects along with a hint. These were the top 2 members occupying the majority of memory (detailed later).

1And the hint was

2

Suspect 1 of accumulation in LinkedList didn’t give me much of the insight, but suspect 2 gave some pointers that it could be something related to database. But this information was not enough to conclude anything.

b. Histogram – Next useful information that I got from memory analyzer is the Histogram.  It shows the number of instances of a particular class and how much memory each one uses.

3

Obviously, the number of byte and char array and memory retained by them doesn’t give information to reach the area in my application causing the problem, which is the culprit code.

One nice feature given by the analyzer is “Group by” option.

4

On grouping by package, it showed the classes by package occupying memory, so that I can look for the classes in my application code.

c. Dominator Tree – The most useful information that I got is from “Dominator Tree”.

It mainly shows the Shallow Heap and Retained Head of a particular object along with the percentage of memory consumed by it.

Shallow Heap of an object is memory consumed by it, that is, its size in the heap.

Retained Heap of an object is the memory kept alive by that object, that is, memory that will be freed when this object is garbage collected.

So what is holding the majority of retained heap is a useful information seen in the Dominator tree.

5

This chart clearly shows that 59.38% of memory is retained by a TaskThread and 35.90% by JDBC4ResultSet.

Another beautiful thing given by this anlyzer is that we can expand each of these entries seen in dominator tree to know each which objects in them are holding majority of the memory.

In my case, it was a linkedlist object in the TaskThread with more than 16 lakhs entries. On further expanding each entry, it was seen that each entry was holding data from a mysql table.

6

So that’s was a big clue, there was something happened in the application which queried the database to get some huge data.

Thanks to the dominator tree again, in the same TaskThread, was an instance of StringBuffer with entry

“select * from XYZ where time > ‘sometime’ order by id”.

The second major memory consumption was by com.mysql.jdbc.JDBC4ResultSet. This contains an arraylist of jdbc object “BufferRow”. On expanding some of the BufferRow objects, it was seen that each BufferRow object was holding row data of XYZ table. More than 46 lakhs objects of BufferRow were seen in the dump.

7.1

Based on this information and the information from Histogram, I was able to reach the class whose method was querying the database with above query.Each row information was set in an object and this object was set into a LinkedList object.

There are lot of other features/information like Inspector, GC roots, etc available in this memory analysis tool. In this particular case (database query causing the issue), the above detailed features were useful for me. With this analysis, the query, the java class and the screen in the application which triggered the issue were identified. After having a discussion with the team and the users, a solution of adding a limit on the database query was implemented as a fix to avoid these situations.

Links:

https://eclipse.org/mat/

https://docs.oracle.com/javase/7/docs/webnotes/tsg/TSG-VM/html/clopts.html

http://help.eclipse.org/luna/index.jsp?topic=%2Forg.eclipse.mat.ui.help%2Fconcepts%2Fshallowretainedheap.html