How to pack Logstash forwarder in Docker containers
DevOps ·Problem
There are plenty of articles out there about how to create Logstash forwarder Docker image. For example here, here and here or use Google to find even more.
So why another one? The answer is very simple. I tried them all and there is a big problem with all of them! If Docker container gets recreated for any reason (system restart, image update or …), all log evens in monitored files will resend again to Logstash server.
Idea
By investigating a lot and reading Logstash forwarder source code, finally I found the missing piece of this puzzle. A quick look into the source code shows us here there is a special file with hardcoded name as .logtsash-forwarder which is being created in the program’s working directory, and actually stores all the information about last harvested points in monitored log files.
What happens is, when Docker container gets destroyed and recreated again these information is lost and forwarder has no clue where to start harvesting again. So what is does? Starts from the beginning.
Solution
Simply store that file in host file system by mounting is as a volume with Docker -v option. But there is another problem! If you look at here in source code again, you’ll notice that it calls os.Rename. What’s wrong with that? It turns out if you only mount this file into Docker image, this call fails. Because in Docker it’s not possible to move files from Docker filesystem into mounted filesystem due to system call restrictions.
Solution
Obviously the solution is to mount a directory instead of file itself. And here is a simple Docker file that does this:
FROM marvambass/oracle-java8
RUN apt-key adv --keyserver pool.sks-keyservers.net --recv-keys 46095ACC8548582C1A2699A9D27D666CD88E42B4 \
&& echo 'deb http://packages.elasticsearch.org/logstashforwarder/debian stable main' | tee /etc/apt/sources.list.d/logstashforwarder.list \
&& apt-get update; apt-get install -y logstash-forwarder
ENV PATH /opt/logstash-forwarder/bin:$PATH
COPY entrypoint.sh /
VOLUME ["/logstash-forwarder-conf", "/certs", /home/logstash]
WORKDIR /home/logstash
ENTRYPOINT ["/entrypoint.sh"]
CMD ["logstash-forwarder", "-config=/etc/logstash-forwarder.conf"]
There are two important points here, in VOLUME you can see /logstash which is exposed and WORKDIR which is pointing to /logstash. Here is a complete code in github. This is actually fork of another forwarder project with these modifications.
How to use it
If you want to persist positions of harvested logs in files simply mount /logstash to host filesystem.
docker run -v /var/cache/logstash:/home/logstash docker-logstash-forwarder ...
And next time you update image or start a new container for forwarder it’ll start harvesting logs from the point it’s left them.
**Update: DOCKERFILE has been updated. See GITHUB commit for more info.