Isaac Rieksts's Java Development Stories

Sunday, March 30, 2014

Intro to Elasticsearch and New Features in version 1.0.x (Snapshot and Restore)

These are the notes from my presentation: http://www.meetup.com/Elasticsearch-Philadelphia/events/164506092/

http://youtu.be/6Idi13rQx1o?t=40m30s

Snapshot and restore is a great way to save your data in case of an accidental deletion or full cluster loss. Below I walk through an example of snapshot and restore. It assumes you have an index philly_energy.

Create back up folder
$ curl -XPUT 'http://localhost:9200/_snapshot/my_backup' -d '{
    "type": "fs",
    "settings": {
        "location": "/tmp/mount/backups/my_backup",
        "compress": true
    }
}'

{"acknowledged":true}

Get backups
$ curl -XGET "localhost:9200/_snapshot/my_backup/_all"

{"snapshots":[]}

Take backup
$ curl -XPUT "localhost:9200/_snapshot/my_backup/snapshot_1" -d '{
    "indices": "philly_energy",
    "ignore_unavailable": "true",
    "include_global_state": false
}'

{"accepted":true}

Get backups
$ curl -XGET "localhost:9200/_snapshot/my_backup/_all"

{"snapshots":[{"snapshot":"snapshot_1","indices":["philly_energy"],"state":"SUCCESS","start_time":"2014-03-28T02:11:07.740Z","start_time_in_millis":1395972667740,"end_time":"2014-03-28T02:11:07.882Z","end_time_in_millis":1395972667882,"duration_in_millis":142,"failures":[],"shards":{"total":5,"failed":0,"successful":5}}]}

Drop index
$ curl -X DELETE 'http://localhost:9200/philly_energy'

{"acknowledged":true}

Restore
$ curl -XPOST "localhost:9200/_snapshot/my_backup/snapshot_1/_restore"

{"accepted":true}

Intro to Elasticsearch and New Features in version 1.0.x (Cat API)

These are the notes from my presentation: http://www.meetup.com/Elasticsearch-Philadelphia/events/164506092/

http://youtu.be/6Idi13rQx1o?t=38m40s

The cat API provides simple ways to monitor your cluster without needed to parse JSON

$ curl 'localhost:9200/_cat/nodes?h=ip,port,heapPercent,name'
127.0.0.1 9300 4 Red Wolf

$ curl 'localhost:9200/_cat/shards'
philly_energy          4 p STARTED    64 97.5kb 127.0.0.1 Red Wolf
philly_energy          4 r UNASSIGNED
philly_energy          0 p STARTED    51 79.3kb 127.0.0.1 Red Wolf
philly_energy          0 r UNASSIGNED
philly_energy          3 p STARTED    42   65kb 127.0.0.1 Red Wolf
philly_energy          3 r UNASSIGNED
philly_energy          1 p STARTED    54 83.6kb 127.0.0.1 Red Wolf
philly_energy          1 r UNASSIGNED
philly_energy          2 p STARTED    48   75kb 127.0.0.1 Red Wolf
philly_energy          2 r UNASSIGNED
park_maintenance       2 p STARTED     0    99b 127.0.0.1 Red Wolf
park_maintenance       2 r UNASSIGNED
park_maintenance       0 p STARTED     0    99b 127.0.0.1 Red Wolf
park_maintenance       0 r UNASSIGNED
park_maintenance       3 p STARTED     4 3.4kb 127.0.0.1 Red Wolf
park_maintenance       3 r UNASSIGNED
park_maintenance       1 p STARTED     0    99b 127.0.0.1 Red Wolf
park_maintenance       1 r UNASSIGNED
park_maintenance       4 p STARTED     4 3.4kb 127.0.0.1 Red Wolf
park_maintenance       4 r UNASSIGNED
park_maintenance_plain 2 p STARTED     0    99b 127.0.0.1 Red Wolf
park_maintenance_plain 2 r UNASSIGNED
park_maintenance_plain 0 p STARTED     0    99b 127.0.0.1 Red Wolf
park_maintenance_plain 0 r UNASSIGNED
park_maintenance_plain 3 p STARTED     1 3.2kb 127.0.0.1 Red Wolf
park_maintenance_plain 3 r UNASSIGNED
park_maintenance_plain 1 p STARTED     0    99b 127.0.0.1 Red Wolf
park_maintenance_plain 1 r UNASSIGNED
park_maintenance_plain 4 p STARTED     1 3.2kb 127.0.0.1 Red Wolf
park_maintenance_plain 4 r UNASSIGNED

Given the format of the cat api it makes it grep friendly

$ curl 'localhost:9200/_cat/shards' | grep park
% Total    % Received % Xferd Average Speed   Time    Time     Time Current
                                 Dload Upload   Total   Spent    Left Speed
100 2040 100 2040    0     0   131k      0 --:--:-- --:--:-- --:--:-- 142k
park_maintenance       2 p STARTED     0    99b 127.0.0.1 Red Wolf
park_maintenance       2 r UNASSIGNED
park_maintenance       0 p STARTED     0    99b 127.0.0.1 Red Wolf
park_maintenance       0 r UNASSIGNED
park_maintenance       3 p STARTED     4 3.4kb 127.0.0.1 Red Wolf
park_maintenance       3 r UNASSIGNED
park_maintenance       1 p STARTED     0    99b 127.0.0.1 Red Wolf
park_maintenance       1 r UNASSIGNED
park_maintenance       4 p STARTED     4 3.4kb 127.0.0.1 Red Wolf
park_maintenance       4 r UNASSIGNED
park_maintenance_plain 2 p STARTED     0    99b 127.0.0.1 Red Wolf
park_maintenance_plain 2 r UNASSIGNED
park_maintenance_plain 0 p STARTED     0    99b 127.0.0.1 Red Wolf
park_maintenance_plain 0 r UNASSIGNED
park_maintenance_plain 3 p STARTED     1 3.2kb 127.0.0.1 Red Wolf
park_maintenance_plain 3 r UNASSIGNED
park_maintenance_plain 1 p STARTED     0    99b 127.0.0.1 Red Wolf
park_maintenance_plain 1 r UNASSIGNED
park_maintenance_plain 4 p STARTED     1 3.2kb 127.0.0.1 Red Wolf
park_maintenance_plain 4 r UNASSIGNED

Monday, August 12, 2013

Cassandra Summit videos

The presentations from the Cassandra Summit in San Francisco are posted here: http://www.planetcassandra.org/blog/post/cassandra-summit-2013---use-cases-and-technical-presentations

I listed some of my favourites below.

First would be my own presentation: "Crossing the Chasm - SQL to NoSQL": http://www.youtube.com/watch?v=wIjvZRLUXsE

Here is an awesome presentation on debugging Cassandra: "In Case Of Emergency, Break Glass": http://www.youtube.com/watch?v=7lY-33_Hn0Q

Titan is an up and coming graph database: "Distributed Graph Computing with Titan and Faunus": http://www.youtube.com/watch?v=tYAWhBHQiXA

Spotify talks about lessons learned using Cassandra: "How Not to Use Cassandra": http://www.youtube.com/watch?v=0u-EKJBPrj8

Sunday, March 24, 2013

Storm Dependency

Hello, I am Isaac Rieksts a Software Developer.

Today, I wanted to talk briefly about how I worked around a dependency conflict when including storm in a webservice.

Storm 0.8.2 has a direct dependency on commons-io 1.4, but includes compojure 1.1.3, which includes ring-core 1.1.5, which is dependent on commons-io 2.1. This caused a problem when a module I wrote pulled in commons-io 2.1 from an internal module. My module did not include storm.

So the deployment ended up choosing commons-io 1.4 to deploy and got an error: java.lang.NoSuchMethodError: org.apache.commons.io.FileUtils.write(Ljava/io/File;Ljava/lang/CharSequence;Z)

Looks like this

Deployment bundler 0.0.1
- webservice module 0.0.1
    - my module 0.0.1
      - internal module 0.0.1
      - commons-io 2.1
    - storm 0.8.2
    - commons-io 1.4
    - compojure 1.1.3
      - ring-core 1.1.5
      - commons-io 2.1

To fix this: I managed commons-io at the web container level:

<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>2.1</version>
</dependency>

<dependency>
<groupId>storm</groupId>
<artifactId>storm</artifactId>
<version>0.8.2</version>
<exclusions>
    <exclusion>
      <artifactId>commons-io</artifactId>
      <groupId>commons-io</groupId>
    </exclusion>
</exclusions>
</dependency>

Deployment bundler 0.0.1
- webservice module 0.0.1
- commons-io 2.1
    - my module 0.0.1
      - internal module 0.0.1
      - commons-io 2.1
    - storm 0.8.2 (Exclude commons-io)
    - compojure 1.1.3
      - ring-core 1.1.5

Now the deployment works fine with no problems.