Run datafusion-examples when start CI

[smallzhongfeng]

Is your feature request related to a problem or challenge?

Currently, we have not verified the accuracy of the test classes in the datafusion-example directory, so we should test it while running ci. Reduce the probability of coding errors

Describe the solution you'd like

The current idea may be relatively simple. I will create two directories, one named external_storage and one named flight_sql. External_storage will contain test cases that require external storage, and flight_sql will contain flight-related test cases. When I run my tests I skip these two directories.

Describe alternatives you've considered

No response

Additional context

No response

[smallzhongfeng]

cc @alamb let me know what you think. Thanks

[devinjdangelo]

This sounds like a good idea to me, but will take a little work in at least one case...

The example showing writing to S3 won't work in CI out of the box since it requires specifying a bucket and credentials with write access. Public write access is not appropriate here, so we would need a set up to securely pass creds to the ci pipeline.

We could also just skip this one example.

[smallzhongfeng]

This sounds like a good idea to me, but will take a little work in at least one case...

The example showing writing to S3 won't work in CI out of the box since it requires specifying a bucket and credentials with write access. Public write access is not appropriate here, so we would need a set up to securely pass creds to the ci pipeline.

We could also just skip this one example.

Yes, I also think I should skip these tests that rely on external storage.

[alamb]

External_storage will contain test cases that require external storage, and flight_sql will contain flight-related test cases. When I run my tests I skip these two directories.

Another option would be to use crate features -- and have two new features like flight and external_storage that default to on, but are not enabled in CI

[smallzhongfeng]

External_storage will contain test cases that require external storage, and flight_sql will contain flight-related test cases. When I run my tests I skip these two directories.

Another option would be to use crate features -- and have two new features like flight and external_storage that default to on, but are not enabled in CI

Sounds great! I try the way you said, but it seems fail. It throw

  consider adding a `main` function to `datafusion-examples/examples/dataframe-to-s3.rs`

I don’t know if I understand what you mean. I added the feature to the test class dataframe-to-s3.rs. like this:

#[cfg(feature = "external_service")]
#[tokio::main]
async fn main() -> Result<()> {
    // create local execution context
    let ctx = SessionContext::new();

...

It couldn't find the specific runtime. I was wondering if I need to encapsulate some logic into a method and then define the feature on this method.