How to: stream through a pipe with python

    tail -f logfile | python email_lines.py

Where email_lines.py is essentially

    for line in stdin.readlines():

        email(line)

Now, there was one major thing that was wrong with it, and that was that it didn’t work. A line would come in, and it wouldn’t get emailed out. Turns out both the pipe and stdin.readlines() were buffering the stream so that only when there was an EOF or like 1k of data would the for loop continue.

To fix this, I added the -u option to python (unbuffered mode) and rewrote email_lines.py to use readline() in a while loop instead of readlines(). So that became

    tail -f logfile | python -u email_lines.py

and

    while 1:

        line = stdin.readline()

        if not line: break

        email(line)

Anyhow, that was that but along the way I learned one bonus thing: tail -f follows a file descriptor by default, but -F will read from a file path, and is resistent to files being renamed. Which turns out to be important because our logs are rotated on a regular basis!

Cheers,
Danny