README.md

FiFoFile v1.0.0

A class that makes it easy to read and write lines in FiFo files (named pipes). This class was created to open a FiFo file and keep reading that file like a tail -f. Its use is not recommended for reading a single line.

A FiFo file does not generate I/O consumption or occupy disk space. Using a FiFo file, you can read tens of thousands of lines per second indefinitely. For sending logs to a FIFO file, we recommend using syslog-ng.

The FiFoFile class detects when syslog is restarted and automatically reopens the FiFo file, preventing the application from trying to read a file that has been closed and increasing CPU consumption.

Installation

pip install fifofile

Usage

Imagine a scenario where syslog-ng must send log lines to a FiFo file and you need to read those lines sent to that FiFo file.

• Create a FiFo file:

mkfifo /var/log/my_fifo_file.fifo
chmod 666 /var/log/my_fifo_file.fifo

• Configure syslog-ng (rsyslog or syslog) to send a copy of your application's logs directly to this fifo file.

  • syslog-ng: https://support.oneidentity.com/technical-documents/syslog-ng-open-source-edition/3.19/administration-guide/41#TOPIC-1094617
  • rsyslog: https://www.rsyslog.com/doc/configuration/modules/ompipe.html
  • syslog: https://medium.com/@mezgani/pipes-in-syslog-f10ea02a00d2

• Use the FifoFile class to read these lines in real time

from fifofile import FiFoFile

fifo = FiFoFile('/var/log/my_fifo_file.fifo')
counter = 0
for line in fifo.readline():
    counter += 1
    print(line)
    if counter == 50:
        fifo.stop_reading() # is important to use this method to EXIT THE GENERATOR and CLOSE THE FIFO FILE.
        print(">>>>>>> STOPPED AS REQUESTED")
        break

or

from fifofile import FiFoFile

with FiFoFile('/var/log/my_fifo_file.fifo') as fifo: # When you exit this block, the FiFo file is closed.
    counter = 0
    try:
        for line in fifo.readline(): # keep reading indefinitely
            print(line) # do some cool stuff with the line
    except:
        fifo.stop_reading()

Methods

• __init__(self, fifo_file_path:str, create_if_not_exists:bool=False, create_mode:str='0o666', polling_timeout:float=1.0)

  Constructor method for the FiFoFile class.

  • fifo_file_path (str): The path to the FIFO file.
  • create_if_not_exists (bool, optional): If True, creates the FIFO file if it doesn't exist. Defaults to False.
  • create_mode (str, optional): The file mode to use when creating the FIFO file. Defaults to '0o666'.
  • polling_timeout (float, optional): The timeout value for polling the FIFO file. Defaults to 1.0.

• create_fifo_file(fifo_file_path:str, create_mode:str='0o666', raise_if_exists:bool=False) -> bool

  Static method to create a fifo file. You don't need to instantiate the class to use it.

  • fifo_file_path (str): The path to the FIFO file.
  • create_mode (str, optional): The file mode to use when creating the FIFO file. Defaults to '0o666'.
  • raise_if_exists (bool, optional): If True, raises an exception if the file already exists. Defaults to False.

• is_fifo_file(fifo_file_path:str) -> bool

  Static method to check if a file is a fifo file. You don't need to instantiate the class to use it.

  • fifo_file_path (str): The path to the file.

• stop_reading()

  Method to stop the reading process. Important to use this method to EXIT THE GENERATOR and CLOSE THE FIFO FILE.

• writeline(self, line:str, flush:bool=True)

  Method to write a line to the fifo file. This method opens the FiFo, writes the line, and closes the FiFo.

  • line (str): The line to write to the fifo file.
  • flush (bool, optional): If True, flushes the fifo file after writing. Defaults to True.

• read(self, size:int) -> Generator[Any, Any, Any]

  Method to read a line with a fixed size from the fifo file and return it as a generator.

  • size (int): The maximum number of characters to read.

• readline(self, strip_line:bool=True) -> Generator[str, None, None]

  Method to read a line (until "\n") from the fifo file and return it as a generator.

  • strip_line (bool, optional): If True, strips leading and trailing whitespace from the line. Defaults to True.

• __enter__(self)

  Method to support the with statement.

• __exit__(self, exc_type, exc_value, traceback)

  Method to support the with statement.

To Do List

• Add asyncio support

Sugestions, feedbacks, bugs...

Open an issue or e-mail me: ricardoabuchaim at gmail.com