How to: stream through a pipe with python

This came up when I was trying to mail out lines appended to the apache error log as they came in. The essence of what I wanted to do could be written succinctly as:

    tail -f logfile | python
Where is essentially
    for line in stdin.readlines():
Now, there was one major thing that was wrong with it, and that was that it didn’t work. A line would come in, and it wouldn’t get emailed out. Turns out both the pipe and stdin.readlines() were buffering the stream so that only when there was an EOF or like 1k of data would the for loop continue.
To fix this, I added the -u option to python (unbuffered mode) and rewrote to use readline() in a while loop instead of readlines(). So that became
    tail -f logfile | python -u
    while 1:
        line = stdin.readline()
        if not line: break
Anyhow, that was that but along the way I learned one bonus thing: tail -f follows a file descriptor by default, but -F will read from a file path, and is resistent to files being renamed. Which turns out to be important because our logs are rotated on a regular basis!



Similar Articles

The World's Most Powerful Mobile Data Collection Platform

Start a FREE 30-day CommCare trial today. No credit card required.

Get Started

Learn More

Get the latest news delivered
straight to your inbox