No instrumentation, no performance tuning!

This is my first time to use grape to build an api service, grape repo has more than 2k watchers, but I'm surprised there is no existing newrelic grape suppport, I just found some gists to do it, and this blog post gave me the idea to add newrelic instrument as grape middleware, but it's not the standard way newrelic recommends.

So I released newrelic-grape gem to help you integrate newrelic into grape.

What you need to do is

require "newrelic-grape"
require "rpm_contrib"

and monitor the performance on newrelic.

I'm helping my customer build a high performance api service these weeks, we are close to release, but when I did load test this Wednesday, I found the memory kept growing when I sent traffic and never went down, it was obviously a memory leak.

Lucky is I can reproduce the memory leak on my local machine, so I can detect it easily. Our api service is simple, only contains model layer (AR and redis) and api layer (based on grape). At first, I disabled model layer, but memory leak was still there, so I was pretty sure the leak was in api layer.

Memory leak is always not easy to find, especially when I'm not sure where it is, in my own code or some dependent libraries I used. I need some tools' help.

First, I used heap_dump to dump the ruby heap memory after sending 10 minutes' traffic, and searched the keywords used in our repository, I noticed every request path string resided in memory, why? Was there an array or hash used them? heap_dump can't answer me.

Then I tried ruby 1.9.3 ObjectSpace to find more info. I changed Grape::API.call behavior, printing live objects for each request.

def call_with_gc(env)
  GC.start
  result = call_without_gc(env)
  p ObjectSpace.count_objects
  result
end

The followings are parts of the result

{:TOTAL=>331126, :FREE=>218067, :T_OBJECT=>3339, :T_CLASS=>3394, :T_MODULE=>474, :T_FLOAT=>195, :T_STRING=>55324, :T_REGEXP=>1135, :T_ARRAY=>20188, :T_HASH=>926, :T_STRUCT=>125, :T_BIGNUM=>22, :T_FILE=>4, :T_DATA=>16011, :T_MATCH=>13, :T_COMPLEX=>1, :T_RATIONAL=>33, :T_NODE=>11273, :T_ICLASS=>602}
[23153:INFO] 2012-12-16 21:59:55 :: Status: 200, Content-Length: 19, Response Time: 42.43ms
{:TOTAL=>331126, :FREE=>218066, :T_OBJECT=>3339, :T_CLASS=>3394, :T_MODULE=>474, :T_FLOAT=>195, :T_STRING=>55325, :T_REGEXP=>1135, :T_ARRAY=>20188, :T_HASH=>926, :T_STRUCT=>125, :T_BIGNUM=>22, :T_FILE=>4, :T_DATA=>16011, :T_MATCH=>13, :T_COMPLEX=>1, :T_RATIONAL=>33, :T_NODE=>11273, :T_ICLASS=>602}
[23153:INFO] 2012-12-16 21:59:56 :: Status: 200, Content-Length: 20, Response Time: 43.29ms
{:TOTAL=>331126, :FREE=>218065, :T_OBJECT=>3339, :T_CLASS=>3394, :T_MODULE=>474, :T_FLOAT=>195, :T_STRING=>55326, :T_REGEXP=>1135, :T_ARRAY=>20188, :T_HASH=>926, :T_STRUCT=>125, :T_BIGNUM=>22, :T_FILE=>4, :T_DATA=>16011, :T_MATCH=>13, :T_COMPLEX=>1, :T_RATIONAL=>33, :T_NODE=>11273, :T_ICLASS=>602}
[23153:INFO] 2012-12-16 21:59:57 :: Status: 200, Content-Length: 20, Response Time: 45.74ms

As you can see, every request, there was a string couldn't be garbage collected, but I still didn't know where it was. I commented my logic code in api layer, just returned an empty json, and string leak still existed, then I went to grape source code, commented the code in Grape::API#call method, updated as following code

def call(env)
  [200, {}, ""]
end

After that, the string memory leak disappeared, It was a strong possibility that memory leak was in grape gem.

Next thing was pretty easy, tried to replace grape middleware one by one, grape middleware has 3 methods, call!, before and after, will be called in every request, I replaced all of them to figure out leak.

Finally, I found it's method format_from_extension in grape Formatter middleware caused memory leak, it genereate a symbol no matter if there is an extension in the request path, e.g.

if requesting /v1/blog/post/1, it will create symbol :"/v1/blog/posts/1"

if requesting /v1/blog/post/2, it will create symbol :"/v1/blog/posts/2"

......

In case you don't know, symbol won't be garbage collected in ruby, so every time it got a request path different then before, it created a symbol in memory which won't be garbage collected.

Problem detected, solution is here.

In conclusion, be careful to ruby symbol, do not convert any non controlled string to symbol.

TL;DR: OpenFeint gets 40% performance improvement after migrating to JRuby from REE.

About OpenFeint

OpenFeint was the largest mobile social gaming platform in the world, It was acquired by GREE for $104 million last year, and a new global platform is building to replace OpenFeint. It is still one of the biggest rails applications, with hundreds of thousands API calls per minute.

OpenFeint platform is using rails 2.3.14 and was running on ree 1.8.7.

Why try JRuby

My main job is to improve the performance and scalability of OpenFeint platform. This April, I attended Railsconf at Austin, there was a panel discussion talking about real world rails apps, speakers came from New Relic, Zendesk, Groupon, etc. They use the similar achitecture like us, ree 1.8.7, rails 2.3, mysql, memcached, redis, rabbitmq and so on. They all complained the slow gc of ruby 1.8.7, so did we. After that, there are 2 jruby sessions interested me.

• Up and to the right - how Spiceworks is scaling 200 million requests per month, they shown how they migrate their rails app to jruby and got 20% performance improvement.
• Complex Made Simple: Sleep Better with Torquebox, it introduced torquebox, a ruby application server that is build on JRuby and JBoss AS 7.

When I went back to hotel, I googled something about jruby performance and found torquebox performance benchmark, it looked pretty exciting. At that time I decided to try jruby on OpenFeint platform.

Note: you probably know new relic and zendesk have already migrated to ruby 1.9.

Quick and dirty performance test with JRuby

I always prefer doing the performance test by myself rather than blindly believing the performance benchmark online. So the first thing I want to do was to do a quick performance test with JRuby on OpenFeint platform.

It was expected that OpenFeint platform couldn't work with JRuby. To quickly verfiy if JRuby could give us a great performance improvement, I fixed incompatible ruby gems, like adding jruby-openssl gem, removing SystemTimer gems and using activerecord-jdbcmysql-adapter instead of mysql gem. I also did some dirty hacks, e.g. I disabled database sharding, background job and other non working parts, just want to do a quick performance test. Then I deployed app to one of our qa servers, the result of quick performance test is as follows

• response time of ree + passenger is 331ms
• response time of jruby + torquebox is 51.5ms

I was shocked that JRuby is so fast, that made it easy to persuade manager to migrate OpenFeint platform to JRuby.

Note: our qa environment is quite different to production environment, databases are shared between qa servers, but memcached, redis, rabbitmq and app server are working together in one host, and ree on qa server didn't do any gc tuning.

JRuby migration strategy

After the quick performance test, JRuby looked very promising, then I'm allowed to focus my work on JRuby migration. Before I tell you how we migrate to JRuby, please let me give you a short introduction about what OpenFeint platform uses

• load balancer servers with nginxes.
• app servers with nginx + passenger.
• memcached servers for caches.
• redis servers for feature flags, high score caches, device mapper, etc.
• mysql servers for data storage.
• uses rabbitmq server and workling servers to handle background jobs.

Of course OpenFeint platform uses other servers for cron job, performance test, continuous integration, full text search, log analytics, etc.

To handle the massive requests, OpenFeint platform splits app and databse servers into different pools according to different functionalities.

Each app pool is isolated, they don't know each other. Load balancer servers decide sending requests to which pool according to the request urls. Each pool will connect to all db servers, e.g. high score app servers will fetch high score info from high score dbs and fetch user/game info from core dbs.

Considering that we don't have experienced java ops and we only have 1 or 2 qas can involve in, it is a big risk to migrate the whole OpenFeint platform to JRuby. So I decide to do JRuby migrate one app pool by one app pool.

The advantage of migration one pool by one pool is it allows OpenFeint gets the JRuby's speed earlier, 1 or 2 qas are enough to promise app is working correctly for one pool, ops can setup jruby environment and tune the jvm performance on one pool's hosts to accumulate jruby experience.

The disadvantage is we have to promise OpenFeint platform is working well on both REE and JRuby, running app with REE on some pools and running app with JRuby on other pools.

Note: only load balancers and mysql servers are dedicated servers, others are VPS.

Fix incompatible gems

The most problems for migrating a rails app to JRuby are incompatible gems, like c extensions gems or some non thread-safe ruby gems. I encountered 2 incompatible gems that wasted my time.

1. typhoeus, it is one of the fastest http client ruby gems, it's a c extenion gem, we used it to synchronize data between OpenFeint platform and the new global platform. The official document says it is built with FFI and is ready for use with any Ruby implementation. But during performance test, I found it always crashed the JVM after running about 1 hour. According to the crash log, I fixed a missing attach_function here, but it didn't help. I ended up using net-http-persistent in JRuby while using typhoeus in REE. From performance test, I surprisingly found JRuby + net-http-persistent isn't slower than REE + JRbuy.

2. memcached, it is the fastest memcached client ruby gems, it's also a c extension gem. At first I used jruby-memcache-client, but jruby-memcache-client uses Base64 to encode/decode value, which can't work with memcached gem together. Then I chose dalli which supports both REE and JRuby, but it uses different hash and distribution algorithms, which causes too much cache misses on production. I searched some other jruby memcached clients, but none of them are compatible with memcached gem, I ended up writing jruby-memcached gem by myself based on spymemcached. I wrote a post about this gem before, check it out here.

Enable threadsafe

By default, threadsafe is disabled in rails 2.3.14, which means every requests are locked by Rack::Lock, it's not a big deal when running in multi-processes servers, like unicorn or passenger, but it loses the JRuby's natvie multi-threads power. So make sure you enable the threadsafe when migrating to JRuby.

Enabling threadsafe means rails won't automatically load libraries under lib/ directory, you have to load them by yourselves.

Enabling threadsafe also means you must consider thread safety seriously. OpenFeint platform uses long-running threads to communicate with scribe, there is a eager loaded global queue and a lazy loaded thread for each process, when doing performance test with JRuby + Torquebox, sometimes it will genereate several lazy loaded threads, and finally cause memory leak. The solution is to eager load the long running thread.

Pass all tests

It's a common sense that you must have good coverage unit, functional and integration tests before doing a big migration. When all tests were passed, I was confident to go further.

Note: JRuby always eat much more memory to run memory, for openfeint platform, I have to allocate 2 GB memory

JRUBY_OPTS=-J-Xmx2g jruby --client -S bundle exec rake test

Pick up a JRuby server

There are 4 JRuby servers that I can choose

• Trinidad, built on JRuby::Rack and Tomcat.
• Torquebox, built on JBoss AS.
• Mizuno, built on Jetty.
• Puma, a new ruby web server built for concurrency.

Puma depends on rack ~> 1.2 but rails 2.3.14 depends on ~> 1.1.0, so I can't try Puma for OpenFeint platform.

I chose Torquebox from the other 3 servers, the reasons are as follows.

1. Torquebox runs faster than Trinidad and Mizuno according to our own performance test, I think this is bacause Torquebox is mostly written by Java while other servers are written by Ruby. 2. Some Torquebox core team members are paid by Red Hat to work on Torquebox project, that means we can get better supports. 3. Torquebox project is very actively developing, and always keeps up with latest JBoss AS server and JRuby.

Note: Recently I replaced torquebox with torquebox-lite, which is a smaller, web-only version of Torquebox, you can easily add other jboss submodules when necessary.

Monitor JVM

Running on JVM is quite different than running on REE, you probably face some new issues, like memory leak and thread safety. We uses New Relic to monitor response time, throughput, etc., but it doesn't help to monitor jvm heap / non heap memory and thread stacks. Fortunately we also use scout to monitor our servers, scout provides JMX Monitoring plugin which collects the memory usage of jvm. It is okay for production so far, but we will use zabbix for better monitoring in the future.

In Java world, there are a lot of monitor tools. Command tools like jstat, jstack and jmap, graphical tools like jconsole and visualvm, you can easily get the heap / non heap memory usage, gc stats, each thead stack trace, etc.

It's really important to monitor JVM when doing performance / stress test, it can help you find out memory leak and thread safe issues before running on production. Here are 2 examples.

1. memory leak, I noticed that heap memory (both edge and old) reached 100% during stress test. Although no OutOfMemoryError raised, it was definitely a memory leak. I used jmap to dump all heap memory and read them by Eclipse MAT, here is the result.

It's a typical memory leak, objects in container can't be gabarge collected.

2. thread safe, I also found the db connection pool in activerecord 2.3.14 is not thread safe. The throughput will decline after running a long time, I used jstack to dump all threads stack trace and saw most of threads are locked in connection_pool as follows.

"http--127.0.0.1-8180-1" daemon prio=10 tid=0x00007f4a17609800 nid=0x725a in Object.wait() [0x0000000049dfc000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
  at java.lang.Object.wait(Native Method)
  - waiting on <0x0000000704f40e18> (a org.jruby.libraries.ThreadLibrary$ConditionVariable)
  ......
  at rubyjit.ActiveRecord::ConnectionAdapters::ConnectionPool#checkout_0978F3C1EFB2CBFA2CD717B12DA76E3113CD78B7.block_1$RUBY$__file__(/home/deploy/rails_apps/  openfeint_platform/shared/bundle/jruby/1.8/gems/activerecord-2.3.14/lib/active_record/connection_adapters/abstract/connection_pool.rb:192)
  at rubyjit$ActiveRecord::ConnectionAdapters::ConnectionPool#checkout_0978F3C1EFB2CBFA2CD717B12DA76E3113CD78B7$block_1$RUBY$__file__.                          call(rubyjit$ActiveRecord::ConnectionAdapters::ConnectionPool#checkout_0978F3C1EFB2CBFA2CD717B12DA76E3113CD78B7$block_1$RUBY$__file__:65535)
  ......

But the count of http threads is equal to the count of db connections, no thread should be locked. Considering our situation, I added a monkey patch to connection_pool with one db connection per thread. It's not perfect but works well.

Tune JVM performance

There are several jvm settings you should set for JRuby performance.

1. Xms and Xmx, when we hot deployed app to Torquebox by touching -knob.yml.dodeploy, it took more than 20 minutes to complete, which was unacceptable, after discussing with Torqeubox support team, I knew default value for Xms is 64m and Xmx is 256m, they are too small, then I increased them to 2g, it took only 100 seconds to hot deploy. The root cause is hot deployment will increase memory a lot, which causing lots of full GCs.

2. CodeCache, when we do the performance test, I found response time suddenly jumped after running a few minutes, the torquebox log told me "CodeCache is full. Compiler has been disabled." CodeCache is a part of non heap memory in Hopspot JVM, it's 64m by default, so I increased it to 256m by setting -XX:ReservedCodeCacheSize=256m, then I don't see the response time jump anymore.

There are a lot of JVM parameters you can tune for your application, talk and learn from some Java experts.

Performance / Stress test

I mentioned I already did a quick performance test, but it didn't make a big sense, because qa and production have different environments. So this time I did performance / stress tests on a reserved host, which has the exactly same environments with production servers, connecting to production database, memcache and redis servers.

Here are test results for actions in one pool.

JRuby is much faster than REE 1.8.7 and Ruby 1.9.3 in both read and write actions. It's promising we can get a big performance improvement on production.

Make sure you run your stress tests multiple times and run long time, some memory leak and thread safety issues are not reproduced every time or not occurred in a short time.

Note: REE in reserved host is already optimized with twitter's settings.

Deployment strategy

Everything was ready, it was time to think about deployment strategy.

In Java world, you can deploy an app by packaging your source code into a war file and copying the war package to app server. We can do the same thing with JRuby, but it will break our existing capistrano deployment script.

We kept existing capistrano deployment script except deploy:restart task, replacing

touch tmp/restart.txt

with

touch /opt/torquebox/current/jboss/standalone/deployments/openfeint_platform-knob.yml.dodeploy

Torquebox will detect openfeint_platform-knob.yml.dodeploy, undeploy old openfeint_platform and deploy new openfeint_platform, works very similar to passenger. But I found everytime we redeploy app, the non heap memory will jump a lot and the app will be super slow (multiple times slower than usual) during redeployment process.

So I decided to deploy app by restarting jboss instead of hot deployment.

sudo /etc/init.d/jboss-as-standalone restart

It solved memory issue, mitigated the slow requests, but introduced a new issue, it will lost the requests during restarting jboss. The solution we used is rolling restart to provide zero downtime deployment, e.g. we have 3 app servers A, B, C

1. tell load balancers stop sending http requests to server A.
2. restart jboss on server A.
3. tell load balancers resend http resquests to server A when jboss on server A is ready.

And restart server B and C one by one following the above steps.

So far, it works perfect, no memory jump and no request lost.

JRuby on production

Finally we successfully migrated to JRuby on production and the response time dropped a lot.

It was about 40% performance improvement, although it was expected, I was still very excited. Actually after fully warming up, it run even faster than you see on the figure.

The following is the response time comparing to ree's 1 week ago.

This is the successful migration for one pool on OpenFeint platform, we have already migrated 5 pools to JRuby, all got ~ 40% performance improvement. I'm still working on the rest pools' migration and looking forward to replacing all OpenFeint servers to JRuby.

Some JRuby servers have been running on OpenFeint platform for more than 2 months, they are running stably and much faster than before according to New Relic's weekly report.

Further

Java 7 introduced invokedynamic feature, a lot of people said enabling invokedynamic made JRuby 1.7 run much faster, closer to Java speed. But I'm failed tn enable invokedynamic feature with Torquebox, saw the following error

18:29:03,515 ERROR [org.torquebox.core.runtime] (Thread-71) Error during execution: ENV['RAILS_ROOT']=RACK_ROOT
ENV['RAILS_ENV']=RACK_ENV
require %q(org/torquebox/web/rails/boot)
: org.jruby.exceptions.RaiseException: (LoadError) load error: haml/buffer -- java.lang.NoClassDefFoundError: org/jruby/runtime/ThreadContext
     at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:1010) [jruby.jar:]
     at ActiveSupport::Dependencies::Loadable.require(/home/deploy/rails_apps/openfeint_platform/shared/bundle/jruby/1.8/gems/activesupport-2.3.14/lib/active_support/dependencies.rb:182)

Torquebox team is trying to fix this issue, I will definitely enable invokedynamic with new Torquebox release, and am looking forward to another big performance improvement.

Some Resources

If you join the JRuby world, the first thing you need to do is to follow Charles Nutter on twitter, he is one of the JRuby core team members and always shares a lot of JRuby knowledge. Also check out his presentations to get latest JRuby features and benchmarks. JRuby wiki pages are helpful to learn everything about JRuby.

At the end, please allow me to thank JRuby and Torquebox team for providing such great things and thank Gree for allowing me to share the knowledge.

switch_user provides a convenient way to switch current user that speeds up your development and reproduce user specified error on production.

Today switch_user gem 0.9.0 is released, all thanks to Luke Cowell.

He is a collaborator of switch_user gem, and did a great job for 0.9.0 gem.

1. did lots of refactors.
2. added unit tests.
3. made switch_user a rails engine.

check out the changelog here.

I also updated switch_user example to use switch_user 0.9.0.

This is my new post on jrubytips. It teaches you how to achieve zero downtime deployment for jruby servers.

http://jrubytips.com/posts/5-zero-downtime-deployment

This is my new post on jrubytips. It tells you the jvm CodeCache which may affect your server performance.

http://jrubytips.com/posts/4-set-proper-value-for-codecache

4 months ago, I released newrelic-workling gem, which helps us montior the performance of background jobs. We used it to find out a GC performance issue. But we still have some cron jobs, who call rake tasks, running in the black box.

So I created a new project newrelic-rake that adds newrelic instrument for rake tasks. Now when I go to the newrelic, I can see the rake tasks listed in Background tasks section, it shows me the average execution time and call count for all rake tasks.

I can also see the performance breakdown for each rake task.

This rake task probably needs to use persistence net http or some c extension http client, and reduce the GC calls.

It's really important to do monitor first, then do optimize.

More precisely I want to say allocating as less objects as you can, rubyzip is just an example.

We have a background job compressing webui assets, uploading to S3, so mobile sdk can download assets to update webui dynamically.

After iphone5 and ios6 came to the market, we received much more webui requests than before, it was expected, but our background job couldn't consume so much asynchronous messages. We could easily scale out by adding more background job servers, but I decided diving deeply into webui job to see if I could speed it up and increase throughput.

Thank newrelic for providing great monitoring service, I saw the webui job took averagely 725ms to complete a webui job, and 80% time was taken by GC calls, WTF. Instead of blaming ruby gc, I blamed our bad code.

I noticed that we used rubyzip to compress webui assets, it was the root reason to cause so much GC.

def create(path, files)
  Zip::ZipFile.open(path, Zip::ZipFile::CREATE) do |z|
    files.each do |file|
      source_path = "#{Rails.root}/public/webui/#{file}"
      expand_dirs(file).each do |dir|
        begin
          z.mkdir dir
        rescue Errno::EEXIST
        end
      end
      z.add file, source_path
    end
  end
end

It sucks, all files are reading and compressing in ruby VM, too many objects are allocated, then cause several GC calls. So I tried to use shell zip command instead of rubyzip.

I did an experiment between rubyzip and shell zip. The followings are code examples.

require 'zip/zip'
GC::Profiler.enable
before_stats = ObjectSpace.count_objects
start = Time.now
Zip::ZipFile.open("test.zip", Zip::ZipFile::CREATE) do |z|
  Dir["**/*"].each do |file|
    z.add file, file
  end
end
puts "Total time: #{Time.now - start}"
after_stats = ObjectSpace.count_objects
puts "[GC Stats] #{before_stats[:FREE] - after_stats[:FREE]} new allocated objects."

# Total time: 0.75344
# [GC Stats] 718691 new allocated objects.

GC::Profiler.enable
before_stats = ObjectSpace.count_objects
start = Time.now
files = Dir["**/*"].map { |file| file unless File.directory?(file) }
`zip test.zip #{files.join(" ")}`
puts "Total time: #{Time.now - start}"
after_stats = ObjectSpace.count_objects
puts "[GC Stats] #{before_stats[:FREE] - after_stats[:FREE]} new
allocated objects."

# Total time: 0.349816
# [GC Stats] 2269 new allocated objects.

As you can see, rubyzip allocates > 700k objects for reading and compressing, and it also takes more than double time to finish the script, shell zip command is a much better solution. So I replaced rubyzip with shell zip in our product.

def create(path, files)
  `cd #{Rails.root}/public/webui && zip #{path} #{files.join(' ')}`
end

After deploying to background job server, I see a big performance improved, it takes only 218ms for webui job to finish, and only 28% time is taken by GC calls. The throughput is also increased from 44cpm to 64cpm, and it can keep up with the webui asyncrhonous messages, we don't need to add more servers, money saved. :-)

So keep in mind, allocating less objects means less GC calls, also means better performance.

Updated: zip_ruby gem gives a similar speed of shell zip command.

require 'zipruby'
GC::Profiler.enable
before_stats = ObjectSpace.count_objects
start = Time.now
Zip::Archive.open("test.zip", Zip::CREATE) do |z|
  Dir["**/*"].each do |file|
    z.add_file file, file
  end
end
puts "Total time: #{Time.now - start}"
after_stats = ObjectSpace.count_objects
puts "[GC Stats] #{before_stats[:FREE] - after_stats[:FREE]} new allocated objects."

# Total time: 0.367729
# [GC Stats] 1116 new allocated objects.

By default, when you deploy your application by capistrano git, it will clone the repository with entire history on production server, but it's meaningless. You should never go to production host and check git log, instead you just need latest code on production host.

With your application grows, git clone with entire history may take a bit longer time than you expected. The following is the time spent with fully cloning.

$ time git clone git@github.com:railsbp/rails-bestpractices.com.git
Cloning into 'rails-bestpractices.com'...
remote: Counting objects: 11438, done.
remote: Compressing objects: 100% (3915/3915), done.
remote: Total 11438 (delta 7012), reused 11277 (delta 6886)
Receiving objects: 100% (11438/11438), 5.52 MiB | 127 KiB/s, done.
Resolving deltas: 100% (7012/7012), done.
git clone git@github.com:railsbp/rails-bestpractices.com.git  0.55s user 0.26s system 1% cpu 55.275 total

But if clone with depth 1, it's finished much faster since there is only 1 revision fetched.

$ time git clone --depth 1 git@github.com:railsbp/rails-bestpractices.com.git
Cloning into 'rails-bestpractices.com'...
remote: Counting objects: 1635, done.
remote: Compressing objects: 100% (1243/1243), done.
remote: Total 1635 (delta 265), reused 1367 (delta 189)
Receiving objects: 100% (1635/1635), 3.02 MiB | 134 KiB/s, done.
Resolving deltas: 100% (265/265), done.
git clone --depth 1 git@github.com:railsbp/rails-bestpractices.com.git  0.24s user 0.17s system 1% cpu 34.236 total

It's time to apply this on your capistrano file to speed up your deployment.

set :scm, :git
set :git_shallow_clone, 1

Warning: git_shallow_clone can't work with branch.

This is my new post on jrubytips. It tells what threadsafe is in rails, and what're the benefits with threadsafe in jruby server.

http://jrubytips.com/posts/3-enable-threadsafe-for-rails