CENTRALIZED LOGS: A GREAT TOOL TO ANALYZE AND SOLVE PRODUCTION ISSUES

6 months have passed since we first released MUS.

MUS is an Android/iOS/web application for listening to music that gives a relevant place to domestic artists content, that sometimes are hard to find in the bast international catalogs, or that haven’t been reached by the digital inclusion yet.

(If you are interested to learn more about MUS Success Case)

Since the first bits were released to the users, we have been improving the app and solving different issues, and a crucial piece for achieving this has been the logs from all of our web and application servers.

This is not news: developers love(?) to read their own logs… so?

The thing is that our platform is pretty much mounted in AWS Beanstalk containers (which as you may or may not know, instances are taken down and up without notice by AWS, as the gods of the cloud lovely decide it is time to replace an instance).

This of course is well handled by AWS (without causing outage in our app), but our logs got lost when an EC2 instance is replaced by another. This encouraged us to start pushing the logs out of the EC2 instances into a centralized log server (and also because we were having about 22 EC2 instances to monitor, that means a lot of logs).

To achieve this, we have been using the elastic ELK stack (elasticsearch, logstash and kibana) and we are loving the results. We have managed to get all logs in there, and we can search on a simple web interface (kibana) for a specific tag or text, filtering by server, by time range, by device, by app version, by piece of code or class, or whatever you need (this of course after some heavy work on the logstash parser).

But besides the log searches, as we had sustained growth of active users, we have been working on making simple dashboards and graphs to provide to the management people with some interesting numbers about active users, services response time, cache hit ratio for proxies, and many other things that the commercial people loves to analyze.

If you want to get all your logs organized and ready to be dug, here it goes a nice tutorial on installing ELK stack on Ubuntu (a little outdated now since elastic released newer versions, but thanks to digitalocean anyway).

If you are interested to learn more about MUS, also a Success Case of One Single Sign One.

We are BigCheese, an engineer in-house team.