Advanced GORM: Performance, Customization, and Monitoring #s2gx

Speaker: Burt Beckwith, SpringSource

Overview

  • demo of potential performance issues with mapped collections in GORM
  • using the hibernate 2nd-level cache
  • monitoring and managing 2nd-level caches
  • app info plugin

Standard Grails One-to-Many

  • library has many visits
  • visit class has person name and date, with backreference to library

What's the problem?

  • hasMany = [visits:Visit] creates a set
  • sets guarantee uniqueness
  • adding to the set required loading all instances from the database to guarantee uniqueness, even if you know the item is unique
  • likewise for a mapped list–lists don't guarantee uniqueness, but they do guarantee order, so they still have to pull all records from the db to get the order right
  • you get a false sense of security since it's lazy-loaded; only partially helpful
  • works fine in development when you only have a few visits, but imagine when you deploy to production and you have 1,000,000 visits and want to add one more
  • risk of artificial optimistic locking exceptions; altering a mapped collection bumps the version, so simultaneous visit creations can break but shouldn't

What's the Solution?

  • don't use collections
  • instead of visit belonging to a library, visit HAS a library
  • different syntax for persisting a visit
  • no cascading; to delete a library you need to delete its visits first in a transactional service method
  • you also lose your collection so you can't do library.visits.size(), etc. but you can still use dynamic finders,which is better anyway since you're only pulling what you need

Standard Grails Many-to-Many

  • user has many roles, role has many users
  • problem is that if all new users are granted a particular role, you get into scaling issues quickly
  • with many to many you have an intermediate table with pointers to both tables and can map the join table
  • the belongsTo in a many to many can go in either class since it's bidirectional
    • but this is the problem since the same amount of data will get loaded either way
  • more efficient to treat kind of like one-to-many and create the user, then grant the role
    • this way you're adding/deleting single records in a single table due to existence of a domain class describing the relationship
  • important to have well-defined equals() and hashCode() methods in your domain classes, as well as implement serializable so you can use second level caching
  • wind up with user.addToRole() or role.addToUsers()
  • no cascading like before–have to manage this yourself

So never used mapped collections?

  • no, you need to examine each case
  • standard approach is fine if the collections are reasonably small–for both sides in the case of many to many
  • the collections will contain proxies, so they're smaller than real isntances until initialized, but still a memory concern
  • great example of something that's convenient and easy out of the box, but when it becomes a problem, you just do it a different way

Using Hibernate 2nd-level Caching

  • great, but have to  be careful because it can burn you in the same way that a query cache in a database can bite you
  • great candidates for caching–anything that's read only and doesn't change often
  • can overuse cache to the point where you're spending more cycles flushing and aren't saving yourself any db traffic–can actually make things worse than just hitting the db

Caching Usage Notes

  • 1st level cache is the hibernate session itself
  • get is always cached
  • can significantly reduce db load by keeping instances in memory
  • can be distributed between multiple servers to let one instance load from the db and share updated instances, avoiding extra db trips
  • "cache true" creates a read/write cache, best for read-mostly object since frequently updated objects will result in excessive cache invalidation (and network traffic when distributed)
  • "cache usage: 'read only'" creates a read-only cache, best for lookup data (countries, zip codes, etc.) that never change

Query Cache

  • can set cacheable true on all the query options; this caches the query and you can grab class instances from this

Hibernate query cache considered harmful?

  • most queries are not good candidates for caching; must be same query with same parameters
  • updates to domain classes will pessimistically flush all potentially affected cache results
  • DomainClass.list() is a decent candidate if there aren't any (or many) updates and the total number isn't huge
  • great blog post by alex miller at http://tech.puredanger.com/2009/07/10/hibernate-query-cache

2nd Level Cache API

  • evict one instance: sessionFactory.evict(DomainClass, id)
  • can get stats (hits/misses, etc.)
  • can look at the stats and get a good sense of whether or not you're caching effectively
  • e.g. if miss count is high then your cache strategy isn't effective
  • appinfo plugin gives you tons of information about what's going on in your app, what's going on in hibernate, etc.

Groovy Web Services, Part I: REST #s2gx

Speaker: Ken Kousen – Kousen IT, Inc.

  • currently working on book for Manning: "Java and Groovy: The Sweet Spots"
  • "Java is really good for tools and infrastructure. Groovy is good for pretty much everything else."

Two Flavors of Web Services

  • SOAP based
    • SOAP wrapper for payload
    • much like an XML API on a system
    • makes header elements available
    • lots of automatic code generation–tools are very mature
      • stubs, skeletons, proxies, etc. are all written for you
      • sprinkle annotations into your codebase to get a web service out of it
  • REST based
    • everything is an addressable resource (uri)
    • leverage http transport
    • uniform interface (get, post, put, delete)
      • a lot of rest web services available today aren't truly restful–e.g. amazon web services where you have to specify the operation name
      • just invoking methods over the web using URLs, but the request type doesn't matter
  • REST -> Representational State Transfer
    • term coined by Roy Fielding in PhD thesis (2000), "Architectural Styles and Design of Network-Based Software Architectures" (available in HTML online; very readable and thought-provoking)
    • one of the founders of the Apache Software Foundation
    • on the team that came up with the spec for HTTP
  • Idempotent
    • repeated requests give the same result
    • get, put, and delete requests are meant to be idempotent
    • POST is not meant to be idempotent
    • e.g. on put–if you're supposed to get the same result every time, the assumption is that you'd know the uri you're applying this to every time
      • the ID will be in the request somehow
      • assumption here is that you know what the ID is going to be before you do the insert
      • having the ID ahead of time probably isn't going to happen in the real world
    • most people wind up doing POST for inserts and PUT for updates
    • "safe" is another term that comes into play–requests do not change the state of the server
      • GET requests should never change the state of the server
  • Content negotiation
    • each resource can have multiple representaitons
      • e.g. xml or json?
    • request will state which representation it wants
  • RESTful client
    1. asemble url with query string
    2. select the http request verb
    3. select the content type
    4. transmit request
    5. process results
  • Steps for rest above are more cumbersome than SOAP since SOAP does a lot of this for you. With REST you have to deal with all of this yourself.
    • where groovy comes into play, is that it's augmented a lot of these classes to make this easier
    • groovy also shines when you get the results back

EXAMPLE: Google Geocoder

  • The Google Geocoding API — converts address info for latitude, longitude
  • Base URL is http://maps.googleapis.com/maps/api/geocode/output?parameters
    • output is XML or JSON
    • parameters include
      • url encoded physical address
      • sensor = true | false (indicates whether or not the request is coming from a GPS-enabled device)
  • Groovy makes it easy to assemble a query string
    • map of parameters, then collect and join, then deal with the xml


def url = base + [sensor:false, address:[loc.street, loc.city, loc.state].collect { v -> URLEncoder.encode(v, 'UTF-8') }.join(',+')].collect {k,v -> "$k=$v"}.join('&')

def response = new XmlSlurper().parse(url)

  • simple read-only web services are a natural fit for REST, even if a lot of other web services in your organization are SOAP
  • get, post, put, and delete also map very naturally to sql verbs, so simple data management apps are natural for rest
  • content negotiation based on URL, e.g. http://…/xml for xml, http://…/json for json

Twitter API

  • closer to true REST
  • in the middle of changing their API again
  • have been supporting basic authentication with base64 encoding for years, which isn't encrypted
  • twitter API is http://dev.twitter.com/doc
  • twitter is oauth only now

HTTP Clients

  • simple http client in groovy — java.net.URL class
  • get is easy, just access the url
  • def xml = new URL(base).text — returns the text of the entire page
  • post, put, and delete in groovy are similar to java

Building a RESTful Service

  • design your URL strategy
    • what are URLs going to mean, what are http verbs going to mean
  • sample url strategy
    • http://…/songs
      • get returns all songs
      • post adds a new song
      • put, delete not used on this url
    • http://…/songs/id
      • get — return a specific song
      • post — not used
      • put — update song with given id
      • delete — remove song with given id

Groovlet

  • groovy servlet — groovy class that responds to an http request
  • easy to configure
  • provides access to the http methods
  • markup builder called html
  • set up the groovy servlet in web.xml
  • showing code for the various operations in the groovlet
  • main tools we have in groovy for restful web services are the xml slurper and markup builder–makes all this very simple

What does Java bring to the table?

  • one of groovy's principles is to not reinvent stuff that's available in java
  • inevitable that there's a performance penalty for using groovy
  • nice thing is you can mix and match groovy with java and use what's appropriate for individual portions of the application
  • JSR311 – JAX-RS
  • currently JAX-WS is part of Java EE 5 but also part of Java SE 1.6
  • specification for JAX-RS — part of Java EE 6

Libraries for HTTP

  • apache http client
  • groovy http builder
  • rest client available in the groovy http builder
  • typical for groovy: take a java library and make it easier/build on it

Grails

  • grails apps make REST easy
  • can set up mappings to map urls to controllers/methods — this is built in
  • xml marhsalling is dead simple in grails — just use "render foo as XML"
    • works great for xml that doesn't go too deep
    • can always use the markup builder

Downside to REST

  • No WSDL, therefore no proxy generation tools
    • all hand-written code for the plumbing
  • WADL: Web Application Description Language
    • attempt to create WSDL for REST
    • slowly gaining acceptance but not common yet

JAX-RS

  • attempt to do for REST what JAX-WS does for SOAP
  • Jersey — reference implementation for JAX-RS
  • annotation based

Conclusions

  • groovy makes it very easy to build URLs with query strings, invoke urls and get responses, and parse xml response
  • groovy works with JAX-RS reference implementation

Location of Grails Webapp When Launched from SpringSource Tool Suite

I'm having trouble getting SpringSource Tool Suite to compile and deploy a Java class located in a Grails project's src/java directory when I run the application from within STS. This is particularly odd given that I have a ServletFilter in this same location (albeit a different package) that gets compiled just fine, so I decided to dig into things a bit and figure out where STS deploys projects when you do grails run-app. This way I could at least see whether or not the class was getting deployed and if my class not found exceptions were related to some other issue.

Turns out if you watch the console as the app launches you get a big clue. Well, the answer actually. I'm on Ubuntu so in my case, my project was deployed to /home/mwoodward/.grails/1.2.1/projects/PROJECT_NAME/resources, so I poked around in there to discover that it even creates the package for my Java class but alas, there's no compiled class in the package.

At least I confirmed I'm not losing my mind. Now to solve the mystery of why STS isn't compiling and deploying that class.

March GroovyMag Available — Includes My “Grails For Switchers” Article

Media_httpwwwgroovyma_qcrpj

The March edition of GroovyMag is available and includes an article I wrote entitled “Grails for Switchers,” which is my account of what I learned while first learning Grails.

If you’re at all interested in Groovy or Grails you need to subscribe to GroovyMag. It’s a great way to learn more about Groovy and Grails and keep up with what’s going on in the Groovy/Grails community.

Resolving CSS Issues With Grails UI Plugin

I'm working on another Grails application and am using the fantastic Grails UI plugin for a lot of the UI controls. Grails UI is a really nice Grails-friendly wrapper around the YUI components and includes things like a dialog box, calendar controls, a rich text editor, and a whole lot more. This was my first real foray into using this plugin, so I started with a simple modal dialog box that would show the contact information details for people in a simple list.

The main point of this post is to outline the simple resolution to the CSS issues I was seeing because it took me a while to figure out what was going on, but I thought I'd outline some Grails and Grails UI magic along the way.

First, in order to use the Grails UI plugin you of course have to install it, which is as simple as:

grails install-plugin grails-ui

Next, on any view page on which you wish to use any Grails UI resources, you have to indicate which resources you're going to use on the page. The nice thing about this is it will only load the JavaScript for the specific UI resources you need on each page. In the case of this example I'm only using a dialog box, so I have this line in the head section of my view page:

<gui:resources components="dialog" />

Also in my head section I need to tell Grails I'll be using some AJAX on this page, so I use the javascript tag to load the Prototype library:

<g:javascript library="prototype" />

With Protoype loaded, pulling the contact details to be shown in the dialog box is dead simple using the Grails remoteLink tag:

<g:remoteLink controller="person"
               action="showDetail"
               id="${person.id}"
               update="personDiv"
               onComplete="showPersonDialog();">${person}</g:remoteLink>

If you're not familiar with Grails, what this does is tells Grails to make an AJAX call to the person controller, call the action showDetail, and pass the ID of the person object. We'll see what's returned by the AJAX call in a moment. The update attribute of the remoteLink tag tells Grails what DOM object to update with the results of the AJAX call, and the onComplete attribute indicates a JavaScript function to call when the AJAX call is complete.

If all I was doing was updating a DIV on the page I wouldn't need this, but since I need to show the Grails UI dialog box after I pull the contact details, I need a JavaScript function to handle that, so I added this to the head section of the view page:

<script type="text/javascript">
    function showPersonDialog() {
        GRAILSUI.personDialog.show();
    }
</script>

Next, let's check out the showDetail action in my Person controller to see what it's doing when the AJAX call is made:

def showDetail = {
    def personInstance = Person.get(params.id)
    if (!personInstance) {
        def message = "No person found with ID ${params.id}"
        render(view:"personDetail", model: [message : message])
    } else {
        render(view:"personDetail", model: [personInstance : personInstance])
    }
}

Here's the simple view that's rendered in the controller action above:

<g:if test="${message}">
    <p>${message}</p>
</g:if>

<g:if test="${personInstance}">
    <p>
        <strong>${personInstance.firstName} ${personInstance.lastName}</strong><br />
        ${personInstance.email}<br />
        Phone: ${personInstance.phone}<br />
        Cell: ${personInstance.cell}
    </p>
</g:if>

And finally, here's the code for the Grails UI dialog box and the DIV that is populated with the view above:

<div class="yui-skin-sam">
    <gui:dialog id="personDialog"
                width="400px"
                title="Contact Details"
                draggable="true"
                update="personDiv"
                modal="true">
        <div id="personDiv"></div>
    </gui:dialog>
</div>

This is all pretty straight-forward. What was happening, however, is when the dialog box was shown, there was no CSS being applied to it. The YUI components that are used by the Grails UI plugin have stylesheets associated with them, but when I checked the source code of the rendered page they seemed to be getting included just fine, and as you can see above I wrapped the dialog box in a div with the correct CSS class, which is yui-skin-sam.

At this point it's important to remember that when a Grails page is rendered it uses SiteMesh, which is basically a templating/page decoration framework. As many Grails applications do, I was using a main.gsp layout page, and each individual view page gets woven into this main template.

Therein lies the problem. As I said above this was simple enough in the end but since it took me a while to figure out I thought I'd share. Even though the YUI CSS was being included in the individual view page with the dialog box code on it, for some reason the CSS wasn't getting applied. I decided to experiment and put the yui-skin-sam class in the body tag in my main.gsp layout page, and this solved the problem.

In my case I didn't have any conflicting CSS involved so this solution didn't cause any issues, but if you have other CSS involved and applying a class to the body tag in the main layout page will cause issues, you can add the additional CSS references after the <g:layoutHead /> tag in the main layout page, and this will allow you to override any CSS that came earlier.

With all this in place the Grails UI components are being styled correctly and they're extremely nice additions to any Grails app.

ThirstyHead: Free Webinar: Getting Started with Groovy, Grails, and MySQL (February 18, 2010)

Wednesday, Feb 10, 2010

Free Webinar: Getting Started with Groovy, Grails, and MySQL (February 18, 2010)

On February 18, 2010, join Scott Davis on a Sun/Oracle sponsored webinar: Getting Started with Groovy, Grails, and MySQL. We’ll spend some time working with MySQL from Groovy 1.7 scripts (Sql.eachRow(), Sql.withBatch()). Then we’ll switch gears and show you how easy it is to skin a MySQL database with Grails 1.2. Hope to see you there!

If you’re interested in Grails this would be a great one to attend. Scott’s a great presenter!

InfoQ: Getting Started with Grails, Second Edition – FREE!

Grails is a Java- and Groovy-based web framework that is built for speed. First-time developers are amazed at how quickly you can get a page-centric MVC web site up and running thanks to the scaffolding and convention over configuration that Grails provides. Advanced web developers are often pleasantly surprised at how easy it is to leverage their existing Spring and Hibernate experience.

“Getting Started with Grails” brings you up to speed on this modern web framework. Companies as varied as LinkedIn, Wired, Tropicana, and Taco Bell are all using Grails. Are you ready to get started as well?

The second edition of “Getting Started with Grails” in now available for free on InfoQ, or you can buy the print version for only $22.95. The first edition was great so I’m really looking forward to reading the update.

In Seattle? Into Groovy and Grails? Join the Seattle Groovy/Grails Meetup!

If you're in the Seattle area and interested in Groovy and Grails, make sure and join us at next month's Seattle Groovy/Grails Meetup on February 11. We get together every month at the Elephant & Castle, and it's a casual mix of Groovy, Grails, other technology, and whatever else comes up.

All are welcome so even if you're not using Groovy and Grails yet, come join us and we can all learn from each other. See you there!

Componentix blog – Run long batch processing jobs in Grails without memory leaks

In one of our Grails applications we had to run a number of batch jobs. Nothing unusual and Grails supports it quite well with the excellent Quartz plugin.

But when we deployed application in production, we noticed that after running for some time, it consumed a lot of memory and JVM was spending all the time running garbage collection. The reason for it was that our jobs were quite long-running, taking several hours to complete, and Grails wasn’t really designed for such kind of use case.

But “not designed for” doesn’t mean it’s not possible, of course. Great info about how to clear out some of the things that will eat up memory over time in long-running Grails processes.

Grails 1.2 Released

Grails 1.2 final was released today, and there's tons of great new features as well as numerous under-the-hood improvements that make it a very compelling release.

Since I'm relatively new to Grails I'm still wrapping my head around some of the new stuff, but here are some of the more interesting changes in 1.2 from my perspective:

  • Performance Improvements
    Faster is always better of course, so the first thing that caught my eye was the performance improvements in GSP/SiteMesh and the tag library return types (default is now StreamCharBuffer, but tags can return object values too).
  • Environments in Bootstrapper
    Not that this was a showstopper in previous versions of grails, but having to use a kludge or do a quick check of the state of your database to see whether or not you should crank out test data seemed a bit wrong since environment awareness is available elsewhere in Grails apps. Now that it's in the bootstrapper too we can do things the "right way."
  • Per-Method Transactions in Services
    I haven't run into issues with this yet but I could see them coming, so it's nice that you can now declare at the method level whether or not something should be transactional.
  • Named Query Support
    I'm still debating whether or not I think this one is a good idea, but it's certainly interesting. Named query support allows you to define named queries right in your domain classes, and these named queries can then be executed in the same fashion as dynamic finders. The really intriguing thing is that you can run dynamic finders on your named queries, which is tremendously powerful. On the one hand I'm thinking, "Queries in my domain class? That seems wrong." But on the other hand, when they're looked at as "customized dynamic finders," I suppose it's really no more intrusive than the dynamic finders themselves. And frankly given the power you can get out of this feature it's probably worth the small purity of architecture trade-off.
  • Support for SQL Restrictions in Criteria Builder
    Much as I love GORM, sometimes it's simpler to throw some ad-hoc SQL at a problem. This is now supported in the criteria builder which should make lots of the "grab stuff then grab stuff from that stuff" type code disappear (if that made any sense!).
  • GSPs are Pre-Compiled in WAR Deployments
    Small new feature, huge impact in my opinion. On Tomcat instances where we're running a few other apps (especially on 32-bit Windows servers … don't get me started), we noticed that Grails apps wouldn't start up due to out of memory errors that I strongly suspect were related to this issue. Pre-compiling GSPs reduces the amount of permgen memory space used on deployment.
  • Tomcat is Now the Default Container
    I didn't have a problem with Jetty personally but since we use Tomcat in production it's nice it will be Tomcat in development as well. Note that you can still use Jetty if you like since all the container support is done through plugins. The Tomcat Plugin has a ton of cool additional features like JNDI support in Config.groovy for the embedded Tomcat instance, as well as remote deploy/undeploy scripts. Yes, you can do "grails tomcat deploy" to deploy your app to a remote server. Sweet.
  • Named URL Mappings
    Very handy feature that lets you create URL mappings that may be linked to by name in the link tag/function. For example if you define a URL mapping with the name "fooBar" with attributes of foo and bar, your link tag in a GSP would be <link:fooBar foo="foo" bar="bar">Link to FooBar</link>
  • Improved JSON Builder
    Some nice changes (and an under-the-hood rewrite apparently) here, but note this is a breaking change from previous versions. You can turn on compatibility if you don't want to re-write your code to take advantage of the new simpler syntax.
  • Better Date Parsing
    One of the very few things I've run into in Grails so far that I thought to myself, "this could be simpler," is the way the date picker passes its params to controllers. Now it works like it should have all along.

Much more in this release so be sure and check out the release notes for all the details.

Kudos to everyone involved with this release! Can't wait to upgrade the current app I'm working on and give this all a whirl.