Skip to content

Implementation to collect queryInfo in S3 using presto event listener

Notifications You must be signed in to change notification settings

ankitdixit/prestoeventlistener

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

9 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

prestoeventlistener

Implementation to collect queryInfo in S3 using presto event listener.

  1. Create a directory $PRESTO_HOME/plugin/event-listener and upload the presto-event-listener-1.0-SNAPSHOT.jar there
  2. Create a new file at $PRESTO_HOME/etc/event-listener.properties with following details:
event-listener.name=event-listener
accessKey=** 
secretKey=**
bucket=<bucket-name>
tableLocationKey=<location-in-bucket>
  1. Restart presto server
  2. Create a new table at S3 location using Hive (change the location at the end):
CREATE EXTERNAL TABLE `eventlistener`(
  `queryid` string,
  `query` string,
  `createtime` timestamp,
  `start_time` timestamp,
  `analysistime` bigint,
  `executionstarttime` timestamp,
  `endtime` timestamp,
  `state` string,
  `error` string,
  `inputtables` string,
  `inputcolumns` string,
  `walltime` bigint,
  `cputime` bigint,
  `totalbytes` bigint,
  `totalmemory` double,
  `queryinfojson` string)
PARTITIONED BY (dt date, hr string)
ROW FORMAT DELIMITED
  FIELDS TERMINATED BY '|'
STORED AS INPUTFORMAT
  'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
  'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
  's3://bucket-name/location-in-bucket'
;
  1. Recover partitions using Hive:
ALTER TABLE eventlistener RECOVER PARTITIONS;

About

Implementation to collect queryInfo in S3 using presto event listener

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages