Druid Segment Diskspace Calculator

Recently i have been working with Druid & was trying to come up with the disk space sizing on the historical nodes, as we have to deploy onto remote customer locations, for which we need to come up with machine requests way in advance.

dc

This took me into the world of bitmaps : concise and roaring.

Druid uses concise bitmaps by default and has the option for roaring too.

References:

So after reading a bit i decided to come up with a calculator for Druid Segment Sizing needed on the druid HISTORICAL nodes assuming Concise is used.

You can find the calculator  @ github.com/jaihind213/druid-calculator

PS: Initially i wanted to write a blog post explaining the logic behind my calculator, but was too lazy, so wrote code and included the rationale in the code comments. 🙂

Thank you and hope you find it useful. Feedback is welcome.

 

 

 

 

My Attempt to demystify Data Stores.

Recently, i gave a presentation at Singapore Py-DataMeetup on  “Demystifying Datastores”.

Screen Shot 2017-03-15 at 11.33.54 AM.png

The main motivation of the presentation is to help one understand the various facets of a datatstore and how these facets can help you decide which one to use.

here is the link

 

 

Hack – How 2 add jars 2 springboot classpath with JarLauncher ?

We had a scenario where  we were using the JarLauncher of springboot but had to add additional jars to classpath.

i.e. we would launch the springboot app using the following command

java -jar fat_app.jar

Since springboot when using JarLaunhcer , ignores -classpath or -cp argument of java, our attempt to add a jar via -cp argument fails.

Solution as of now is to modify the pom.xml and change to Properties launcher i.e. change pom.xml as follows

<build> 
    <plugins>
        <plugin>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-maven-plugin</artifactId>
            <configuration>  <!-- added -->
                <layout>ZIP</layout> <!-- to use PropertiesLaunchar -->
            </configuration>
        </plugin>
    </plugins>
</build>

But we DID NOT want to do the above.

We wanted to add a jar, without changing the pom.xml

i.e. continue using the JAR Launcher.

When we use the jar launcher i.e. run via the following command

java -jar fat_app.jar
# The main class in JAR manifest 
# is set to 'org.springframework.boot.loader.JarLauncher'

The solution is to go the old java way and provide the main class yourself:

i.e. the solution is

java \
-cp fat_app.jar \
-Dloader.path=<path_to_your_additional_jars> \
org.springframework.boot.loader.PropertiesLauncher

# It should work now, even though the main class in manifest 
# of fat_app.jar is set to 'org.springframework.boot.loader.JarLauncher'
# and your additional classes will be picked up by springboot

 

#Thanks to my ex colleague @Dapeng for the idea

 

Load Average high due to Ntop using 100% Cpu

So recently while being oncall, i noticed that the load average was very high on my aws ec2 instance.

The first thing i did, was run TOP command: which showed me the following:

Screen Shot 2016-06-15 at 10.49.26 pm

Ntop –  taking 105 % cpu and  500MB  !!!!

Now thats Fishy !!

I googled and found out these two links which suggested that NTOP is broken:

https://forum.pfsense.org/index.php?topic=2554.0

&

https://bugs.launchpad.net/ubuntu/+source/ntop/+bug/461141

So i proceeded to stop ntop, but stop failed, so i just killed it !

kill   -9   <ntop_pid>

and

wallah the load averages returned to normal 🙂

 

Screen Shot 2016-06-15 at 11.37.43 pm

Symptoms:

  • Ntop running for a while.
  • load average slowly increasing
  • increase in network traffic