Add support for additional log drivers

[haircommander]

Currently CRI format is the only one supported. At minimum long term, json-file and journald should also be added. #482 tracks adding splunk as a log driver as well

[ghost]

@haircommander @saschagrunert are there resources (links, code files etc) one can look at to understand this issue much better?

[saschagrunert]

Hey, most of the container log driver abstraction lives in

conmon-rs/conmon-rs/server/src/container_log.rs

Lines 12 to 19 in 634fd10

	pub struct ContainerLog {
	drivers: Vec<LogDriver>,
	}
	#[derive(Debug)]
	enum LogDriver {
	ContainerRuntimeInterface(CriLogger),
	}

Means we would require to adapt this part, the cap n proto interface

conmon-rs/conmon-rs/common/proto/conmon.capnp

Lines 42 to 56 in 634fd10

	struct LogDriver {
	# The type of the log driver.
	type @0 :Type;
	# The filesystem path of the log driver, if required.
	path @1 :Text;
	# The maximum log size in bytes, 0 means unlimited.
	maxSize @2 :UInt64;
	enum Type {
	# The CRI logger, requires `path` to be set.
	containerRuntimeInterface @0;
	}
	}

as well as the client including testing

conmon-rs/pkg/client/client.go

Lines 632 to 652 in 634fd10

	// ContainerLogDriver specifies a selected logging mechanism.
	type ContainerLogDriver struct {
	// Type defines the log driver variant.
	Type LogDriverType
	// Path specifies the filesystem path of the log driver.
	Path string
	// MaxSize is the maximum amount of bytes to be written before rotation.
	// 0 translates to an unlimited size.
	MaxSize uint64
	}
	// LogDriverType specifies available log drivers.
	type LogDriverType int
	const (
	// LogDriverTypeContainerRuntimeInterface is the Kubernetes CRI logger
	// type.
	LogDriverTypeContainerRuntimeInterface LogDriverType = iota
	)

[TimePrinciple]

Hi, I'm interested in this idea.

I've just cargo tested conmon-rs, and I didn't see any UTs in file conmon-rs/conmon-rs/server/src/container_log.rs, thus not quite sure what kind of output is desired.

I wonder the works to be done is to reformat the output CriLogger to JSON and splunk, or to create different ContainerRuntimeInterface for both.

[jellllly420]

Hi, I'm interested in this project.

I'm currently not so familiar with the details in conmon-rs. However, from the structure of the code, I guess the works to be done in this project is to implement different loggers like the CriLogger and add them to the tuple enum LogDriver and at last modify the related cap n proto and go parts. If there is anything wrong, please let me know. :)

Hope to join and finish this project.

[aashi-ihsaa]

@haircommander I have experience in rust and containerization and this issue seems like a good way to go about that. I would like to take this up under LFX term 3. Can you just a bit elaborate on what kind of output is desired

[haircommander]

thanks for your interest @aashi-ihsaa ! for this one, there are three different goals, all of which are going to be on the server side:

• json logger
• journald logger
• (optional) fluentd log driver
• (optional) splunk log driver

the client pieces should easily follow

[octonawish-akcodes]

I am interested in this issue under LFX sep term

[octonawish-akcodes]

@haircommander Can you give some guidance on what to look up for, so that I could've a better understanding of this issue.

[haircommander]

the resulting output will be equivalent log driver behavior to Docker's https://docs.docker.com/config/containers/logging/configure/ for some or all of the aforementioned drivers

[octonawish-akcodes]

Do you have any more resources? I can look up to @haircommander

[haircommander]

what level is your understanding of the ecosystem @octonawish-akcodes ?

[prash240303]

hello @haircommander @saschagrunert , i tried to understand the functions for the criologger and read the docker docs ,and tried to write the same for the json logger (also i couldnt get the crio setup locally to test this out with conmon , so , its just skimming on the codebase i read and tried to understand) ihope things work this way and for journald logger too , also i couldnt understand the client side testing

#[derive(Debug, Default)]
pub struct ContainerLog {
    drivers: Vec<LogDriver>,
}

#[derive(Debug)]
enum LogDriver {
    ContainerRuntimeInterface(CriLogger),
    JsonLogger(JsonLogger), // New JSON logger variant
}

// New JSON logger implementation
#[derive(Debug)]
struct JsonLogger {
    type @0 :Type; 
    path @1 :Text; 
    max_size @2 :UInt64; 
    }

Tho i am not sure with the implementaion of json logger (so left the functions empty)

impl JsonLogger {
    fn new(path: String, max_size: u64) -> Result<Self> { 
        Ok(Self {
            path,
            max_size,
            //function here
        })
    }

    fn init(&mut self) -> Result<()> {
     //json logger initalization
        Ok(())
    }

    fn reopen(&mut self) -> Result<()> {
      //reopen the resources for json logging 
        Ok(())
    }

    fn write<T>(&mut self, _pipe: Pipe, _bytes: T) -> Result<()>
    where
        T: AsyncBufRead + Unpin + Copy,
    {
      //writing json logs
        Ok(())
    }
}

container log implementation impl ContainerLog {
    // ... (existing methods)

    pub fn from(reader: Reader<Owned>) -> Result<SharedContainerLog> {
        let drivers = reader
            .iter()
            .flat_map(|x| -> Result<_> {
                Ok(match x.get_type()? {
                    Type::ContainerRuntimeInterface => {
                        LogDriver::ContainerRuntimeInterface(CriLogger::new(
                            x.get_path()?,
                            if x.get_max_size() > 0 {
                                Some(x.get_max_size() as usize)
                            } else {
                                None
                            },
                        )?)
                    }
                    Type::JsonLogger => {
                        LogDriver::JsonLogger(JsonLogger::new(x.get_path()?)?)
                    }
                })
            })
            .collect();
        Ok(Arc::new(RwLock::new(Self { drivers })))
    }
    
    // ... (existing methods)

    // New method to initialize JSON loggers
    pub async fn init_json_loggers(&mut self) -> Result<()> {
        join_all(
            self.drivers
                .iter_mut()
                .filter_map(|x| {
                    if let LogDriver::JsonLogger(json_logger) = x {
                        Some(json_logger.init())
                    } else {
                        None
                    }
                })
                .collect::<Vec<_>>(),
        )
        .await
        .into_iter()
        .collect::<Result<Vec<_>>>()?;
        Ok(())
    }

    // New method to reopen JSON loggers
    pub async fn reopen_json_loggers(&mut self) -> Result<()> {
        join_all(
            self.drivers
                .iter_mut()
                .filter_map(|x| {
                    if let LogDriver::JsonLogger(json_logger) = x {
                        Some(json_logger.reopen())
                    } else {
                        None
                    }
                })
                .collect::<Vec<_>>(),
        )
        .await
        .into_iter()
        .collect::<Result<Vec<_>>>()?;
        Ok(())
    }

    // New method to write JSON logs
    pub async fn write_json_logs<T>(&mut self, pipe: Pipe, bytes: T) -> Result<()>
    where
        T: AsyncBufRead + Unpin + Copy,
    {
        join_all(
            self.drivers
                .iter_mut()
                .filter_map(|x| {
                    if let LogDriver::JsonLogger(json_logger) = x {
                        Some(json_logger.write(pipe, bytes))
                    } else {
                        None
                    }
                })
                .collect::<Vec<_>>(),
        )
        .await
        .into_iter()
        .collect::<Result<Vec<_>>>()?;
        Ok(())
    }
}

[ItsOrganic]

Hey @haircommander LFX Fall 23 applicant here, I would like to contribute to this project as I have working knowledge of Rust and containerization. We just need to create new separate modules for each and support logs formatting say for JSON we need to format logs data to JSON including metadata and timestamps similarly for Journald and splunk.

[d4v1d03]

Hey there @haircommander,
I've filled the Fall application as of now and would like you to know my proficiency in RUST and container runtime technologies such as K8s,CRI-O and containerd. I also have a solid understanding of runC and its role as a default runtime for Docker and K8s, enabling the execution of containers according to OCI specifications. I was exploring the codebase of conmon-rs and would like to work over it. I've started the work for the JSON driver and would try my best to make a PR in the upcoming week.