Updating my ELK stack with GELF appender

A couple of days ago I wrote about setting up an ELK stack with docker-compose. I did some small changes to the set up, so I thought it’s worth an update.

First change is in the docker-compose.yml to enable logstashs gelf input, move the Log4j socket to port 12202 and add an UDP port forwarding for docker on port 12201:

elasticsearch:
  image: elasticsearch
  ports:
  - 9200:9200
logstash:
  image: logstash:latest
  links:
  - elasticsearch:elasticsearch
  ports:
  - 12201:12201/udp
  - 12202:12202
  command: logstash agent -e 'input { gelf { port => "12201" } log4j { mode => "server" port => "12202"} } output { elasticsearch { hosts => ["elasticsearch"] } }'
kibana:
  image: kibana
  links:
  - elasticsearch:elasticsearch
  ports:
  - 5601:5601
  environment:
  - ELASTICSEARCH_URL=http://elasticsearch:9200

This way, logstash will listen to ports 12201 with the gelf input and port 12202 with the Log4j socket input.

Now I modified the log4j.properties for my application and replaced the Log4j SocketAppender with a biz.paluch.logging.gelf.log4j.GelfLogAppender:

log4j.appender.gelf=biz.paluch.logging.gelf.log4j.GelfLogAppender
log4j.appender.gelf.Threshold=INFO
log4j.appender.gelf.Host=udp:127.0.0.1
log4j.appender.gelf.Port=12201
#log4j.appender.gelf.Version=1.1
#log4j.appender.gelf.Facility=java-test
log4j.appender.gelf.ExtractStackTrace=true
log4j.appender.gelf.FilterStackTrace=true
log4j.appender.gelf.MdcProfiling=true
log4j.appender.gelf.TimestampPattern=yyyy-MM-dd HH:mm:ss,SSSS
log4j.appender.gelf.MaximumMessageSize=8192

# This are static fields
log4j.appender.gelf.AdditionalFields=environment=local

The gelf appender is much more configurable and the search abilities in logstash are now a bit more useful to me. But I have to add another dependency to my application now:

<dependency>
    <groupId>biz.paluch.logging</groupId>
    <artifactId>logstash-gelf</artifactId>
    <version>1.8.0</version>
</dependency>

You can find the documentation at https://github.com/mp911de/logstash-gelf.

Building an ELK stack with docker-compose

Because I have a hard time searching logfiles during development (I like to run everything on DEBUG), I decided to build myself an ELK stack (elasticsearch, logstash and kibana) to throw all my logs into and have a nice UI to search for a special log message.

Fortunately there are official docker images for these tools:

So, everything is easily available, I just needed to figure out how to glue it together.

Because I’m not interested in storing the data over a long period of time, I don’t care about the setup of the elasticsearch engine. When I’m done developing or debugging things, I want to throw away everything and start with a clean environment. So I don’t store anything outside of the docker containers and I don’t want to write any Dockerfiles myself.

The way to go is a simple docker-compose.yml which I can start with a single command and have everything set up to accept log messages from my java applications. So, here we go:

elasticsearch:
  image: elasticsearch
  ports:
  - 9200:9200
logstash:
  image: logstash:latest
  links:
  - elasticsearch:elasticsearch
  ports:
  - 12201:12201
  command: logstash agent --debug -e 'input { log4j { mode => "server" port => "12201"} } output { elasticsearch { hosts => ["elasticsearch"] } stdout {} }'
kibana:
  image: kibana
  links:
  - elasticsearch:elasticsearch
  ports:
  - 5601:5601
  environment:
  - ELASTICSEARCH_URL=http://elasticsearch:9200

As you can see, I only use images – no custom Dockerfile needed. I also put the configuration for logstash and kibana into the docker-compose.yml, so when I use another computer (and I do that often) I can just copy this one file, run docker-compose up, open http://localhost:5601 and get going.

I also exposed port 12201 on the logstash host to be able to send log messages from locally running applications. I modified my log4j.properties and added a new appender named logstash with the following configuration:

log4j.rootLogger=INFO,logstash
...
log4j.appender.logstash=org.apache.log4j.net.SocketAppender
log4j.appender.logstash.port=12201
log4j.appender.logstash.remoteHost=127.0.0.1

The org.apache.log4j.net.SocketAppender comes from the default log4j installation, so no funky external dependencies needed. Just add those lines when deploying locally and the application will log to my ELK stack.

I tried to use docker networks, but logstash acted up and I think it does not particularly like underscores in the elasticsearch hostname. Unfortunately, docker-compose will generate hostnames with underscores, so I had to stick with the old way and use links. If you have any insight how to use a network, please get in touch.

Podcasts I listen to regularly

I’m commuting a lot in the last weeks and will continue to do so for the next couple of months. Over the last years I was on trains pretty often too. One of the things that keep me sane – during these extended periods of time inbetween people talking about all kind of stupid things – are podcasts.

In case you don’t know about podcasts yet, go educate yourself.

If you’re searching for some interesting podcasts or just want to know what I’m listening to, here’s list of my feeds:

It’s not complete, but you can download a full list of my podcast feeds too. The link goes to an OPML file which you can import into your podcast client right away. Or you can use this OPML Browser to see all feeds first.

Kategorien
Sport

Fortschritte und Rückschläge im Training

Seit ein paar Wochen übe ich vermehrt den Handstand, da ich darauf setze, dass in den kommenden Crossfit Open Workouts der Handstand wieder eine Rolle spielen wird. Handstand Pushups werden auf jeden Fall dran kommen, aber ich würde mich nicht wundern, wenn man dieses Jahr auch auf den Händen laufen muss…

Wie auch immer, die 2 Videos unten habe ich bei einem Training gemacht. Einmal funktioniert es schon ganz gut, einmal gar nicht. Das passiert oft, wenn ich eine Übung noch nicht wirklich drin hab. Je mehr ich übe und je besser ich die Technik gelernt habe, desto voraussagbarer wird das Ergebnis. Im Moment befinde ich mich bei Handstand also in der sehr anstrengenden Mitte, in der ich zwar die Grundlagen kann, aber die Ausführung noch nicht so sauber ist, dass ich verlässlich jedes Mal mein Ziel erreichen kann.

Handstand Training

Handstand Training

Naja, noch ein paar Wochen Übung, und dann sollte der Handstand funktionieren. Ich hoffe, dass ich dann auch relativ schnell das Laufen hinbekomme.