hadoop - Create temporary file to load in Hive table using stdout redirection -

i create script load tsv file hive. since .tsv file contains header first have create temporary file without it.

in script.hql have following:

drop table metadata if exists ; create table metadata (     id int,     value string ) row format delimited fields terminated '\t' stored textfile ;  ! tail -n +2 metadata.tsv > tmp_metadata.tsv ;  load data local 'tmp_metadata.tsv' metadata ; 

the problem hive complains > should make redirection new fails , therefore scripts fails.

how can fix this?

1. create new shell script named script.sh , add in shell script:

#!/bin/sh tail -n +2 metadata.tsv > tmp_metadata.tsv hive -v -f ./script.hql 

2. instead of script.hql:

drop table metadata if exists ;

create table metadata (id int,value string) row format delimited fields terminated '\t' stored textfile ;

! tail -n +2 metadata.tsv > tmp_metadata.tsv ;

load data local 'tmp_metadata.tsv' metadata ;

add script.hql:

drop table if exists metadata; create table metadata (     id int,     value string ) row format delimited fields terminated '\t' stored textfile ; load data local inpath 'tmp_metadata.tsv' table metadata ; 

drop table metadata if exists ; not correct. change drop table if exists metadata;

and

load data local 'tmp_metadata.tsv' metadata ; not correct. change load data local inpath 'tmp_metadata.tsv' table metadata ;

3. now, change permission shell script , execute it:

sudo chmod 777 script.sh ./script.sh