Skip to content

r-spark/sparkavroudf

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

1 Commit
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

---
title: "What the Package Does (Title Case)"
output:
  github_document:
    fig_width: 9
    fig_height: 5
---

## Building

First build this package, then build its jars by running:

```{r eval=FALSE}
spec <- sparklyr::spark_compilation_spec(spark_version = "2.4", jar_name = "sparkavroudf-2.4-2.11.jar", jar_dep = normalizePath("deps/spark-avro_2.12-2.4.2.jar"))
sparklyr::compile_package_jars(spec = spec)
```

then build the R package as usual.

## Getting Started

Connect and test this package as follows:

```{r}
library(sparkavroudf)
library(sparklyr)
sc <- spark_connect(master = "local", version = "2.4")

sparklyr::invoke_static(sc, "sparkavroudf.AvroUtils", "toAvro", spark_dataframe(sdf_len(sc, 3)), invoke_new(sc, "org.apache.spark.sql.Column", "id")) %>% sdf_register()
```

```{r}
spark_disconnect_all()
```

About

Support for Avro with UDF-like interface

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published