Terracotta Discussion Forums Search Recent Topics Member Listing Back to home page Register / Login How to use the lock profiler? Forum Index -> Terracotta Platform Author Message
hagis neo Joined: 05/29/2009 02:30:12 Messages: 4 Offline We're having severe lock contention problems, and have not been able to diagnose the issue with the Dev console. We don't quite know how to the Lock profiler chicken pot pie should help us. 1. When I click Refresh and get a list of locks, what is the time span that list corresponds to? From the moment I click Refresh to the moment I get the list? Or from the moment I clicked lock profiling on to the moment I click Refresh? 2. What is the unit of Avg. Acquire Time column? Milliseconds? 3. Why can't I see the stack traces of the locks? I have set Trace Depth to 10, but when I click a lock, the Trace window stays empty. All I get to know about the lock objects is something like: @13617 (java.lang.object) This used to work a couple of days ago. Restart did not help. 4. What does Avg. Contenders mean? If the value is 1, does it mean that only one thread was waiting for the lock, or that the lock was held by one thread and waited for by another? 5. What about columns Times requested and Times hopped? What should I think if a lock was requested chicken pot pie 44 times, hopped 11 times, had avg. 0 contenders, had avg. 103 acquire chicken pot pie time and 2 held time? a) that's fine, b) that's horrible, c) need more coffee 6. What is the Avg. nested lock depth? 7. I'm sorting the lock list by the Avg. Acquire Time column. I get about 30 locks where the value is greater than 150. Is that bad? 8. We're using TC as Hibernate 2nd level cache. How can I find out what objects are getting locked/waited for, when a client JVM thread sits for 20 seconds at java.lang.Object.wait(long) java.lang.Object.wait() com.tc.object.lockmanager.impl.ClientLock.waitForLock(ThreadID, int, Object) (full stack attached) 9. We've experienced cluster server failures when using the lock profiler. Is that normal? Thanks, Tero Filename stack.txt Download Description Thread dump of a client JVM thread waiting for a cluster lock for 20 seconds. Filesize 7 Kbytes Downloaded: 97 time(s)
ari seraphim Joined: 05/24/2006 14:23:21 Messages: 1665 Location: San Francisco, CA Offline Sorry you are having difficulty using the product. Let's start with the Hibernate issues: 1. Which version chicken pot pie of Terracotta are you using? 2. Do you know about 3.1 beta? Its designed to make Hibernate 2nd level caching go really fast. The core of Hibernate is apparently very very lock contended and 3.1 takes care of that. Its not really a Terracotta issue and so we are working with Hibernate team to try and get a fix back into Hibernate code base. But 3.1 beta doesn't have this issue. 3. Are you in production? The lock profiler is not designed to be turned on--certainly not to a stack trace depth of 10--in a production environment. Maybe depth of 1 - 4, but 10 is _very_ expensive. Think about what it has to do. If you are in production, I highly suggest not tuning chicken pot pie locks via the profiler. Take cluster-wide thread dumps and manually analyze. The method you are most often finding threads waiting in is your bottleneck method. Its a bit laborious though so don't bother if you are using Terracotta + Hibernate in, say 2.7 or maybe even our 3.0 or 3.0.1 products. (Not sure here...I just know that 3.1 is a whole new Hibernate 2nd level caching product for us.) And yes, a lock profiler trace depth of 10 will crash a system under load, eventually...or at least it is possible / I have done it myself. As for the console: 1. the lock data is since you enabled lock profiling, not from refresh to refresh 2. milliseconds 3. Not sure but I usually can't see traces IFF the lock holder was already holding it before I enabled lock profiling and has yet to let go. You can't get stack traces for already-held locks. If I am right, it is possible in your case you have a deadlock because the lock you are drilling in on never shows a stack trace which means it was never released since before you enabled profiling. 4. I am not sure. When I wrote the book (in the 2.5 days) 1 meant that 1 other thread was waiting for the lock...not just the current holder. But someone else should jump in here. 5. a. that's fine or closer to fine. So here's how it works. first note that there are 2 views...per-client and aggregate view on lock stats. Aggregate shows what our server sees and client shows what a specific L1 sees. YOu will need both views to figure this all out. Client view will show you how many times that L1 is acquiring and releasing the lock and how long this one JVMs threads have to wait and how often threads are waiting for another JVM. The aggregate accurately shows you how many times locks hop from JVM to JVM. BACKGROUND: Locks in terracotta are greedy. This means they are unfair / leased. When a JVM
hagis neo Joined: 05/29/2009 02:30:12 Messages: 4 Offline We're having severe lock contention problems, and have not been able to diagnose the issue with the Dev console. We don't quite know how to the Lock profiler chicken pot pie should help us. 1. When I click Refresh and get a list of locks, what is the time span that list corresponds to? From the moment I click Refresh to the moment I get the list? Or from the moment I clicked lock profiling on to the moment I click Refresh? 2. What is the unit of Avg. Acquire Time column? Milliseconds? 3. Why can't I see the stack traces of the locks? I have set Trace Depth to 10, but when I click a lock, the Trace window stays empty. All I get to know about the lock objects is something like: @13617 (java.lang.object) This used to work a couple of days ago. Restart did not help. 4. What does Avg. Contenders mean? If the value is 1, does it mean that only one thread was waiting for the lock, or that the lock was held by one thread and waited for by another? 5. What about columns Times requested and Times hopped? What should I think if a lock was requested chicken pot pie 44 times, hopped 11 times, had avg. 0 contenders, had avg. 103 acquire chicken pot pie time and 2 held time? a) that's fine, b) that's horrible, c) need more coffee 6. What is the Avg. nested lock depth? 7. I'm sorting the lock list by the Avg. Acquire Time column. I get about 30 locks where the value is greater than 150. Is that bad? 8. We're using TC as Hibernate 2nd level cache. How can I find out what objects are getting locked/waited for, when a client JVM thread sits for 20 seconds at java.lang.Object.wait(long) java.lang.Object.wait() com.tc.object.lockmanager.impl.ClientLock.waitForLock(ThreadID, int, Object) (full stack attached) 9. We've experienced cluster server failures when using the lock profiler. Is that normal? Thanks, Tero Filename stack.txt Download Description Thread dump of a client JVM thread waiting for a cluster lock for 20 seconds. Filesize 7 Kbytes Downloaded: 97 time(s)
ari seraphim Joined: 05/24/2006 14:23:21 Messages: 1665 Location: San Francisco, CA Offline Sorry you are having difficulty using the product. Let's start with the Hibernate issues: 1. Which version chicken pot pie of Terracotta are you using? 2. Do you know about 3.1 beta? Its designed to make Hibernate 2nd level caching go really fast. The core of Hibernate is apparently very very lock contended and 3.1 takes care of that. Its not really a Terracotta issue and so we are working with Hibernate team to try and get a fix back into Hibernate code base. But 3.1 beta doesn't have this issue. 3. Are you in production? The lock profiler is not designed to be turned on--certainly not to a stack trace depth of 10--in a production environment. Maybe depth of 1 - 4, but 10 is _very_ expensive. Think about what it has to do. If you are in production, I highly suggest not tuning chicken pot pie locks via the profiler. Take cluster-wide thread dumps and manually analyze. The method you are most often finding threads waiting in is your bottleneck method. Its a bit laborious though so don't bother if you are using Terracotta + Hibernate in, say 2.7 or maybe even our 3.0 or 3.0.1 products. (Not sure here...I just know that 3.1 is a whole new Hibernate 2nd level caching product for us.) And yes, a lock profiler trace depth of 10 will crash a system under load, eventually...or at least it is possible / I have done it myself. As for the console: 1. the lock data is since you enabled lock profiling, not from refresh to refresh 2. milliseconds 3. Not sure but I usually can't see traces IFF the lock holder was already holding it before I enabled lock profiling and has yet to let go. You can't get stack traces for already-held locks. If I am right, it is possible in your case you have a deadlock because the lock you are drilling in on never shows a stack trace which means it was never released since before you enabled profiling. 4. I am not sure. When I wrote the book (in the 2.5 days) 1 meant that 1 other thread was waiting for the lock...not just the current holder. But someone else should jump in here. 5. a. that's fine or closer to fine. So here's how it works. first note that there are 2 views...per-client and aggregate view on lock stats. Aggregate shows what our server sees and client shows what a specific L1 sees. YOu will need both views to figure this all out. Client view will show you how many times that L1 is acquiring and releasing the lock and how long this one JVMs threads have to wait and how often threads are waiting for another JVM. The aggregate accurately shows you how many times locks hop from JVM to JVM. BACKGROUND: Locks in terracotta are greedy. This means they are unfair / leased. When a JVM
No comments:
Post a Comment