Cassandra crashed, found suspect objects in MAT analysis

By -

One of the node in a cluster is crashed and when I try to analyze using MAT I see following suspects in the report.

Can any one please help me to understand if there is a problem with Cassandra version,etc?

Version: apache-cassandra-3.11.4 Cassandra Xmx: 13G

java version :1.8.0_45

Last message in Debug log:

INFO  [ScheduledTasks:1] 2022-03-04 21:27:47,182 MessagingService.java:1236 - MUTATION messages were dropped in last 5000 ms: 1 internal and 1 cross node. Mean internal dropped latency: 16314 ms and Mean cross-node dropped latency: 16410 ms
INFO  [ScheduledTasks:1] 2022-03-04 21:27:47,182 MessagingService.java:1236 - REQUEST_RESPONSE messages were dropped in last 5000 ms: 0 internal and 6 cross node. Mean internal dropped latency: 0 ms and Mean cross-node dropped latency: 15855 ms
INFO  [ScheduledTasks:1] 2022-03-04 21:27:47,182 StatusLogger.java:47 - Pool Name                    Active   Pending      Completed   Blocked  All Time Blocked
WARN  [GossipTasks:1] 2022-03-04 21:27:47,431 FailureDetector.java:278 - Not marking nodes down due to local pause of 13948176954 > 5000000000

DEBUG [GossipTasks:1] 2022-03-04 21:27:51,763 FailureDetector.java:284 - Still not marking nodes down due to local pause
DEBUG [GossipTasks:1] 2022-03-04 21:27:51,763 FailureDetector.java:284 - Still not marking nodes down due to local pause
DEBUG [GossipTasks:1] 2022-03-04 21:27:51,763 FailureDetector.java:284 - Still not marking nodes down due to local pause
INFO  [ScheduledTasks:1] 2022-03-04 21:27:51,746 WARN  [Service Thread] 2022-03-04 21:27:53,329 GCInspector.java:282 - G1 Young Generation GC in 1155ms.  G1 Eden Space: 608174080 -> 0; G1 Old Gen: 12492994960 -> 13214415248; G1 Survivor Space: 88080384 -> 29360128;
INFO  [Service Thread] 2022-03-04 21:27:53,329 StatusLogger.java:47 - Pool Name                    Active   Pending      Completed   Blocked  All Time Blocked

Highly appreciate on any tips further to explore?

Suspect 1:

One instance of “org.apache.cassandra.transport.Message$Dispatcher$LegacyFlusher” loaded 
by “sun.misc.Launcher$AppClassLoader @ 0x48062f2e0” occupies 1,856,054,480 (18.04%) 
bytes. The memory is accumulated in one instance of “java.lang.Object[]”, loaded by 
“<system class loader>”, which occupies 1,856,054,136 (18.04%) bytes.

Suspect 2:

One instance of “org.apache.cassandra.transport.Message$Dispatcher$LegacyFlusher” loaded 
by “sun.misc.Launcher$AppClassLoader @ 0x48062f2e0” occupies 1,521,968,760 (14.79%) 
bytes. The memory is accumulated in one instance of “java.lang.Object[]”, loaded by “<system class loader>”, which occupies 1,373,989,312 (13.36%) bytes.

Keywords

org.apache.cassandra.transport.Message$Dispatcher$LegacyFlusher

sun.misc.Launcher$AppClassLoader @ 0x48062f2e0
java.lang.Object[]


Comments

Post a Comment

Popular posts from this blog

Spring Elasticsearch Operations

By -
Elasticsearch Operations Spring Data Elasticsearch uses two interfaces to define the operations that can be called against an Elasticsearch index. These are ElasticsearchOperations and ReactiveElasticsearchOperations. Whereas the first is used with the classic synchronous implementations, the second one uses reactive infrastructure. The default implementations of the interfaces offer: Read/Write mapping support for domain types. A rich query and criteria api. Resource management and Exception translation. ElasticsearchTemplate The ElasticsearchTemplate is an implementation of the ElasticsearchOperations interface using the Transport Client. @Configuration public class TransportClientConfig extends ElasticsearchConfigurationSupport {   @Bean   public Client elasticsearchClient() throws UnknownHostException {                      Settings settings = Settings.builder().put("cluster.name", "elasticsearch")....
Read more

Network Error and Timeout on Authorize.net JS

By -
i am using authorize.net acceptUI.js in order to create a hosted payment form for credit card acceptance. The problem arising now is that, going on the checkout page, I have the following console message: GET https://js.authorize.net/v3/AcceptUI.js net::ERR_NAME_NOT_RESOLVED Which is weird, cause I cannot ping the domain or anything, as my computer cannot resolve the domain. However, this issue is also reported by individual customers and it seems to be sporadically and not happening the entire time, cause i can see other orders going through. Does anyone know how such an error can be caused, or any ideas how it can be resolved? Thanks for your help EDIT: I just checked the A DNS record for js.authorize.net and am getting the following result only: js.authorize.net. 100 IN CNAME js.authorize.net.cdn.cloudflare.net. Doing the same for js.authorize.net.cdn.cloudflare.net. does not return an A Record, unless I specifically instruct to use 1.1.1.1 as DNS server I...
Read more

Object oriented programming concepts (OOPs)

By -
Object-oriented programming (OOP) is a programming paradigm based on the concept of "objects". Re-usability is the main concept of Oops. There are few major principles of object-oriented paradigm. If any programming Language supports this principles that programming language called Object-oriented programming. Below are object oriented programming concepts : 1. Object        Read More about Object     http://theprogrammersfirst.blogspot.in/2016/07/java-object-concept.html
Read more