Hack – How 2 add jars 2 springboot classpath with JarLauncher ?

We had a scenario where  we were using the JarLauncher of springboot but had to add additional jars to classpath.

i.e. we would launch the springboot app using the following command

java -jar fat_app.jar

Since springboot when using JarLaunhcer , ignores -classpath or -cp argument of java, our attempt to add a jar via -cp argument fails.

Solution as of now is to modify the pom.xml and change to Properties launcher i.e. change pom.xml as follows

<build> 
    <plugins>
        <plugin>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-maven-plugin</artifactId>
            <configuration>  <!-- added -->
                <layout>ZIP</layout> <!-- to use PropertiesLaunchar -->
            </configuration>
        </plugin>
    </plugins>
</build>

But we DID NOT want to do the above.

We wanted to add a jar, without changing the pom.xml

i.e. continue using the JAR Launcher.

When we use the jar launcher i.e. run via the following command

java -jar fat_app.jar
# The main class in JAR manifest 
# is set to 'org.springframework.boot.loader.JarLauncher'

The solution is to go the old java way and provide the main class yourself:

i.e. the solution is

java \
-cp fat_app.jar \
-Dloader.path=<path_to_your_additional_jars> \
org.springframework.boot.loader.PropertiesLauncher

# It should work now, even though the main class in manifest 
# of fat_app.jar is set to 'org.springframework.boot.loader.JarLauncher'
# and your additional classes will be picked up by springboot

 

#Thanks to my ex colleague @Dapeng for the idea

 

Advertisements

Load Average high due to Ntop using 100% Cpu

So recently while being oncall, i noticed that the load average was very high on my aws ec2 instance.

The first thing i did, was run TOP command: which showed me the following:

Screen Shot 2016-06-15 at 10.49.26 pm

Ntop –  taking 105 % cpu and  500MB  !!!!

Now thats Fishy !!

I googled and found out these two links which suggested that NTOP is broken:

https://forum.pfsense.org/index.php?topic=2554.0

&

https://bugs.launchpad.net/ubuntu/+source/ntop/+bug/461141

So i proceeded to stop ntop, but stop failed, so i just killed it !

kill   -9   <ntop_pid>

and

wallah the load averages returned to normal 🙂

 

Screen Shot 2016-06-15 at 11.37.43 pm

Symptoms:

  • Ntop running for a while.
  • load average slowly increasing
  • increase in network traffic

 

 

 

 

Graphing Amazon RDS MySQL Metrics with Prometheus & Grafana

Recently the mysql community got an awesome monitoring solution for mysql

with Prometheus & Grafana. The graphs are simply beautiful and really lively.

I started off with this nice post on the mysql performance blog  by Roman Vynar and got the solution up and running very easily.

You can actually monitor Amazon RDS instance with the same steps mentioned in the above post but with a few changes:

 

The monitoring framework consists of 4 components:

  1. Prometheus server on port 9090
  2. Grafana server on port 3000
  3. MySQL  exporter  process which connects to the MySQL Server
  4. Node Exporter process which gets system metrics of the box hosting MySQL

 

Prerequisites:

  1. Create an RDS instance.
  2. create an amazon ec2-instance. (this will host all the 4 components)

Setup:

Component 3 & 4: – Node Exporter & MySQL exporter process :

Amazon _DOES_NOT_ allow us to install anything on the RDS box.

So, I am sorry we will not be able get the System metrics of RDS – please rely on cloudwatch / Rds console for load averages, cpu usage , io etc etc.

So Follow the steps as mentioned in the nice post BUT make the following changes,

  • Install the Node exporter & MySQL exporter processes on the ec-instance ,
    • So the ‘/opt/prometheus/prometheus.yml’ file will look like
    • i.e. you are now monitoring system metrics of the ec2-instance not RDS box !
cat << EOF > /opt/prometheus/prometheus.yml
global:
 scrape_interval: 5s
 evaluation_interval: 5s
scrape_configs:
 - job_name: linux
 target_groups:
 - targets: ['localhost:9100']
 labels:
 alias: db1
 - job_name: mysql
 target_groups:
 - targets: ['localhost:9104']
 labels:
 alias: db1
EOF

But we need to tell the MySQL exporter to pull from RDS endpoint, so the

my.cnf file for MySQL exporter should be as follows:

[root@centos7 prometheus_exporters]# cat << EOF > .my.cnf
[client]
user=prom
password=abc123
host=amazon-rds-instance.amazonaws.com
EOF

 

Component 1 & 2: – Grafana & Prometheus:

Just Follow the steps as mentioned in the nice post .

x———————————————————–x

And walllaaaah…. you should be able graph Amazon RDS metrics 🙂

Screen Shot 2016-05-05 at 4.08.17 pm

 

 

Punch Clock for debugging Apache Storm

I have been working a lot with Transactional topologies of Apache Storm these days.

In the course of my work, i have come up with questions like

  • Why is the topology stuck ?
  • Which part of the topology is the batch stuck at ?

So I came up with an idea based on a punch clock.

Screen Shot 2016-01-13 at 9.47.36 pm

To Find out …

  1. When did the Person enter / exit the office ?
  2. Who is still in office ?

Context Change:

Screen Shot 2016-01-13 at 9.50.33 pm.png

Screen Shot 2016-01-13 at 9.50.52 pm.png

Apply Punch clock to Storm:

Screen Shot 2016-01-13 at 9.50.58 pm

Possible Solution:

Screen Shot 2016-01-13 at 10.08.35 pm

 

Screen Shot 2016-01-13 at 10.03.08 pm

Code:

In the emitBatch method of Partitioned Transactional Spout:

punchCardId = "SPOUT__"+ InetAddress.getLocalHost().getHostAddress()+Thread.currentThread().getId()+"__"+System.currentTimeMillis();
 
PunchClock.getInstance().punchIn(punchCardId); // Punch In 
collector.emit(tuples); // Emit tuple(s) 
PunchClock.getInstance().punchOut(punchCardId); // Punch Out 

—————

Prepare method of Transactional Bolt:

       punchCardId ="Bolt__"+Thread.currentThread().getId()+"__"+System.currentTimeMillis();  //Create Punch Card for txn 

Execute method of Transactional Bolt:

       PunchClock.getInstance().punchIn(punchCardId);      // Punch In

In the finishBatch method of Transactional Bolt:

       PunchClock.getInstance().punchOut(punchCardId);  // Punch Out

—————

Screen Shot 2016-01-13 at 10.06.34 pm

spc

Small Demo:

Screen Shot 2016-01-13 at 5.57.09 pm

Screen Shot 2016-01-13 at 5.57.39 pm

PS: if there are no punch cards available anywhere & topology is stuck, then the problem is probably not your bolts/spout.

Code: https://github.com/jaihind213/storm-punch-clock

Presentation: here

Other Solutions:

(1) logging while entering and exiting

(2) using http://riemann.io/ -> suggested by my friend Angad @Inmobi

Hope you like the idea & hope its useful to you.

Thank you for reading & any feedback is welcome.

 

 

 

The Tricolor in a graph :)

One can always find inspiration randomly – that’s what i believe.

While debugging my Apache Storm topology – i was looking at metrics

using ambari & i noticed something call out to me.

it was the tricolor – the flag of my nation INDIA ==> INSPIRATION.

Screen Shot 2015-12-10 at 4.31.02 pm

#JAIHIND  #GraphArt #inspiration