Sending file logs to Loggly in a Docker container

I'm going to outline how I automate the sending of file logs to Loggly inside my Docker containers. This article is a minor follow-on to my previous article on automated deployments, which gives a good overview of my overall deployment process. In this one I will show you how to use the latest Loggly API to send your file logs across.

I'm going to assume that you are building your docker container using a Dockerfile. If not I recommend doing so, as they provide you ample flexibility and allow you to use version control to track changes to your container setup.

Syslog-ng config

We're going to use syslog-ng - an open source implementation of the syslog protocol - to actually watch the log files for changes and send them to Loggly. We need to configure syslog-ng and tell it what to do. We want to send the actual syslog as well as the log files for our app, which will be running as a server with the container. The below configuration is based on the Loggly docs for syslog-ng:

##############################
## file: syslogng_loggly.conf
##############################
 
source s_syslog {
  file("/var/log/syslog");
};
 
template t_syslog { template("<${PRI}>1 ${ISODATE} ${HOST} ${PROGRAM} ${PID} ${MSGID} [<LOGGLY_TOKEN>@41058 tag=\"syslog\"] $MSG\n");
  template_escape(no);
};
 
destination d_loggly_syslog {
  tcp("logs-01.loggly.com" port(514) template(t_syslog));
};
 
log {
  source(s_syslog);
  destination(d_loggly_syslog);
};
 
 
source s_app {
  file("/var/log/myapp/logfile1");
  file("/var/log/myapp/logfile2");
};
 
template t_app { template("<${PRI}>1 ${ISODATE} ${HOST} ${PROGRAM} ${PID} ${MSGID} [<LOGGLY_TOKEN>@41058 tag=\"app\"] $MSG\n");
  template_escape(no);
};
 
destination d_loggly_app {
  tcp("logs-01.loggly.com" port(514) template(t_app));
};
 
log {
  source(s_app);
  destination(d_loggly_app);
};

In the above configuration I'm watching both /var/log/syslog and two app log files for changes and then sending them to Loggly with the tags syslog and app respectively.

The tagging allows me to easily filter the logs within the Loggly dashboard. The <loggly_token> above should replaced by your own customer authentication token provided by Loggly.</loggly_token>

DOCKERFILE

Somewhere within your Dockerfile you need to install syslog-ng and then set it to use the above configuration file. This can be done as so:

RUN apt-get update  
RUN apt-get install syslog-ng  
ADD syslogng_loggly.conf /etc/syslog-ng/conf.d/loggly.conf  

So you just have to make sure that the syslogng_loggly.conf file is in the same folder as your Dockerfile when doing the build.

And that's it! It's really quite simple. I've successfully been using the above setup in production now for months.