This came up when I was trying to mail out lines appended to the apache error log as they came in. The essence of what I wanted to do could be written succinctly as:
tail -f logfile | python email_lines.py
Where email_lines.py is essentially
for line in stdin.readlines():
Now, there was one major thing that was wrong with it, and that was that it didn’t work. A line would come in, and it wouldn’t get emailed out. Turns out both the pipe and stdin.readlines() were buffering the stream so that only when there was an EOF or like 1k of data would the for loop continue.
To fix this, I added the -u option to python (unbuffered mode) and rewrote email_lines.py to use readline() in a while loop instead of readlines(). So that became
tail -f logfile | python -u email_lines.py
line = stdin.readline()
if not line: break
Anyhow, that was that but along the way I learned one bonus thing: tail -f follows a file descriptor by default, but -F will read from a file path, and is resistent to files being renamed. Which turns out to be important because our logs are rotated on a regular basis!