THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
| Info |
|---|
These functions can be used without GROUP BY as well. |
Return Type | Name(Signature) | Description | OSS |
|---|---|---|---|
bigint | count(*) |
count(expr) |
count(DISTINCT expr[, expr...]) | count(*) - Returns the total number of retrieved rows, including rows containing NULL values. count(expr) - Returns the number of rows for which the supplied expression is non-NULL. count(DISTINCT expr[, expr]) - Returns the number of rows for which the supplied expression(s) are unique and non-NULL. Execution of this can be optimized with hive.optimize.distinct.rewrite. | ||
double | sum(col), sum(DISTINCT col) | Returns the sum of the elements in the group or the sum of the distinct values of the column in the group. | |
double | avg(col), avg(DISTINCT col) | Returns the average of the elements in the group or the average of the distinct values of the column in the group. | |
double | min(col) | Returns the minimum of the column in the group. | |
double | max(col) | Returns the maximum value of the column in the group. | |
double | variance(col), var_pop(col) | Returns the variance of a numeric column in the group. | |
double | var_samp(col) | Returns the unbiased sample variance of a numeric column in the group. | |
double | stddev_pop(col) | Returns the standard deviation of a numeric column in the group. | |
double | stddev_samp(col) | Returns the unbiased sample standard deviation of a numeric column in the group. | |
double | covar_pop(col1, col2) | Returns the population covariance of a pair of numeric columns in the group. | |
double | covar_samp(col1, col2) | Returns the sample covariance of a pair of numeric columns in the group. | |
double | corr(col1, col2) | Returns the Pearson coefficient of correlation of a pair of numeric columns in the group. | |
double | percentile(bigint col, p) | Returns the exact pth percentile of a column in the group (does not work with floating point types). p must be between 0 and 1. NOTE: A true percentile can only be computed for integer values. Use PERCENTILE_APPROX if your input is non-integral. | |
array<double> | percentile(bigint col, array( |
p1 [, |
p2]...)) | Returns the exact percentiles p1, p2, ... of a column in the group (does not work with floating point types). pi must be between 0 and 1. NOTE: A true percentile can only be computed for integer values. Use PERCENTILE_APPROX if your input is non-integral. | ||
double | percentile_approx(double col, p [, B]) | Returns an approximate pth percentile of a numeric column (including floating point types) in the group. The B parameter controls approximation accuracy at the cost of memory. Higher values yield better approximations, and the default is 10,000. When the number of distinct values in col is smaller than B, this gives an exact percentile value. | |
array<double> | percentile_approx(double col, array( |
p1 [, |
p2]...) [, B]) | Same as above, but accepts and returns an array of percentile values instead of a single one. | ||
| double | regr_avgx(independent, dependent) | Equivalent to avg(dependent). | |
| double | regr_avgy(independent, dependent) | Equivalent to avg(independent). | |
| double | regr_count(independent, dependent) | Returns the number of non-null pairs used to fit the linear regression line. | |
| double | regr_intercept(independent, dependent) | Returns the y-intercept of the linear regression line, i.e. the value of b in the equation dependent = a * independent + b. | |
| double | regr_r2(independent, dependent) | Returns the coefficient of determination for the regression. | |
| double | regr_slope(independent, dependent) | Returns the slope of the linear regression line, i.e. the value of an in the equation dependent = a * independent + b. | |
| double | regr_sxx(independent, dependent) | Equivalent to regr_count(independent, dependent) * var_pop(dependent). | |
| double | regr_sxy(independent, dependent) | Equivalent to regr_count(independent, dependent) * covar_pop(independent, dependent). | |
| double | regr_syy(independent, dependent) | Equivalent to regr_count(independent, dependent) * var_pop(independent). | |
array<struct { | histogram_numeric(col, b) | Computes a histogram of a numeric column in the group using b non-uniformly spaced bins. The output is an array of size b of double-valued (x,y) coordinates that represent the bin centers and heights | |
array | collect_set(col) | Returns a set of objects with duplicate elements eliminated. | |
array | collect_list(col) | Returns a list of objects with duplicates. | |
| int | ntile(INTEGER x) | Divides an ordered partition into |
| Tip |
|---|
Most of the UDAFs ignore NULL values. |
...
Row-set columns types | Name(Signature) | Description | OSS |
|---|---|---|---|
T | explode(ARRAY<T> a) | Explodes an array to multiple rows. Returns a row-set with a single column (col), one row for each element from the array. | |
Tkey,Tvalue | explode(MAP<Tkey,Tvalue> MAP<Tkey,Tvalue> m) | Explodes a map to multiple rows. Returns a row-set with two columns (key,value) , one row for each key-value pair from the input map. | |
| int,T | posexplode(ARRAY<T> a) | Explodes an array to multiple rows with an additional positional column ofint type (position of items in the original array, starting with 0). Returns a row-set with two columns (pos,val), one row for each element from the array. | GenericUDTFPosExplode |
T1,...,Tn | inline(ARRAY<STRUCT<f1ARRAY<STRUCT<f1:T1 T1,...,fn:Tn>> fn:Tn>> a) | Explodes an array of structs to multiple rows. Returns a row-set with N columns (N = number of top level elements in the struct), one row per struct from the array. | |
| T1,...,Tn/r | stack(int r,T1 V1 T1 V1,...,Tn Tn/rVn Vn) | Breaks up n values V1,...,Vninto r rows. Each row will have n/r columns. r must be constant. | GenericUDTFStack |
| string1,...,stringn | json_tuple(string jsonStr,stringk1 k1,...,stringkn kn) | Takes JSON string and a set of n keys, and returns a tuple of n values. This is a more efficient version of the get_json_object UDF because it can get multiple keys with just one call. | GenericUDTFJSONTuple |
string string 1,...,stringn | parse_url_tuple(string urlStr,stringp1 p1,...,stringpn pn) | Takes URLstring and a set of n URL parts, and returns a tuple of n values. This is similar to the parse_url() UDF but can extract multiple parts at once out of a URL. Valid part names are HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE, USERINFO, QUERY:<KEY>. |
...
There is no good engine without string manipulation functions. Apache Hive has rich built-instring functions.
Return Type | Name(Signature) | Description | OSS |
|---|---|---|---|
int | ascii(string str) | Returns the numeric value of the first character of str. | |
string | base64(binary bin) | Converts the argument from binary to a base64string. | |
| int | character_length(string str) | Returns the number of UTF-8 characters contained in str. The function char_length is shorthand for this function. | GenericUDFCharacterLength |
| string | chr(bigint|double A) | Returns the ASCII character having the binary equivalent to A. If A is larger than 256 the result is equivalent to chr(A % 256). Example: select chr(88); returns "X". | UDFChr |
string | concat(string|binary A,string|binary B...) | Returns the string or bytes resulting from concatenating the strings or bytes passed in as parameters in order. For example, concat('foo', 'bar') results in 'foobar'. Note that this function can take any number of input strings. | |
array<struct<string,double>> | context_ngrams(array<array<string>>, array<string>,int K,int pf) | Returns the top-k contextual N-grams from a set of tokenized sentences, given a string of "context". See StatisticsAndDataMining for more information. | |
string | concat_ws(string SEP,string A,string B...) | Like concat() above, but with custom separator SEP. | |
string | concat_ws(string SEP, array<string>) | Like concat_ws() above, but taking an array ofstrings. | |
string | decode(binary bin,string charset) | Decodes the first argument into a string using the provided character set (one of 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16'). If either argument is null, the result will also be null. | |
| string | elt( |
N int,str1 string,str2s tring,str3 string,...) | Return string at index number. For example elt(2,'hello','world') returns 'world'. Returns NULL if N is less than 1 or greater than the number of arguments. | ||||||||||||||||
binary | encode(string src,string charset) | Encodes the first argument into a BINARY using the provided character set (one of 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16'). If either argument is null, the result will also be null. | |||||||||||||||
| int | field(val T,val1 T,val2 T,val3 T,...) | Returns the index of val in the val1,val2,val3,... list or 0 if not found. For example field('world','say','hello','world') returns 3. | |||||||||||||||
int | find_in_set(string str,string strList) | Returns the first occurrence of str in strList where strList is a comma-delimited string. Returns null if either argument is null. Returns 0 if the first argument contains any commas. For example, find_in_set('ab', 'abc,b,ab,c,def') returns 3. | |||||||||||||||
string | format_number(number x,int d) | Formats the number X to a format like '#,###,###.##', rounded to D decimal places, and returns the result as a string. If D is 0, the result has no decimal point or fractional part. | |||||||||||||||
string | get_json_object(string json_string,string path) | Extracts JSON object from a JSON string based on JSON path specified, and returns JSON string of the extracted JSON object. It will return null if the input JSON string is invalid.
| |||||||||||||||
boolean | in_file(string str,string filename) | Returns true if the string | |||||||||||||||
int | instr(string str,string substr) | Returns the position of the first occurrence of | |||||||||||||||
int | length(string A) | Returns the length of the string. | |||||||||||||||
int | locate(string substr,string str[,int pos]) | Returns the position of the first occurrence of substr in str after position pos. | |||||||||||||||
string | lower(string A) lcase(string A) | Returns the string resulting from converting all characters of B to lowercase. For example, lower('fOoBaR') results in 'foobar'. | |||||||||||||||
string | lpad(string str,int len,string pad) | Returns str, left-padded with pad to a length of len. If str is longer than len, the return value is shortened to len characters. In the case of an empty padstring, the return value is null. | |||||||||||||||
string | ltrim(string A) | Returns the string resulting from trimming spaces from the beginning(left-hand side) of A. For example, ltrim(' foobar ') results in 'foobar '. | |||||||||||||||
array<struct<string,double>> | ngrams(array<array<string>>,int N,int K,int pf) | Returns the top-k N-grams from a set of tokenized sentences, such as those returned by the sentences() UDAF. See StatisticsAndDataMining for more information. | |||||||||||||||
| int | octet_length(string str) | Returns the number of octets required to hold the string str in UTF-8 encoding. Note that octet_length(str) can be larger than character_length(str). | GenericUDFOctetLength | ||||||||||||||
string | parse_url(string urlString,string partToExtract [,string keyToExtract]) | Returns the specified part from the URL. Valid values for partToExtract include HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE, and USERINFO. For example, parse_url('http://facebook.com/path1/p.php?k1=v1&k2=v2#Ref1', 'HOST') returns 'facebook.com'. Also, a value of a particular key in QUERY can be extracted by providing the key as the third argument, for example, parse_url('http://facebook.com/path1/p.php?k1=v1&k2=v2#Ref1', 'QUERY', 'k1') returns 'v1'. | |||||||||||||||
string | printf(String format, Obj... args) | Returns the input formatted according to printf-style formatstrings. | |||||||||||||||
| string | quote(String text) | Returns the quoted string
| |||||||||||||||
string | regexp_extract(string subject, string pattern, int index) | Returns the string extracted using the pattern. For example, regexp_extract('foothebar', 'foo(.*?)(bar)', 2) returns 'bar.' Note that some care is necessary in using predefined character classes: using '\s' as the second argument will match the letter s; '\\s' is necessary to match whitespace, etc. The 'index' parameter is the Java regex Matcher group() method index. | |||||||||||||||
string | regexp_replace(string INITIAL_STRING, string PATTERN, string REPLACEMENT) | Returns the string resulting from replacing all substrings in INITIAL_STRING that match the java regular expression syntax defined in PATTERN with instances of REPLACEMENT. For example, regexp_replace("foobar", "oo|ar", "") returns 'fb.' Note that some care is necessary in using predefined character classes: using '\s' as the second argument will match the letter s; '\\s' is necessary to match whitespace, etc. | |||||||||||||||
string | repeat(string str, int n) | Repeats | |||||||||||||||
| string | replace(string A, string OLD, string NEW) | Returns the string A with all non-overlapping occurrences of OLD replaced with NEW. Example: select replace("ababab", "abab", "Z"); returns "Zab". | UDFReplace | ||||||||||||||
string | reverse(string A) | Returns the reversed string. | |||||||||||||||
string | rpad(string str,int len,string pad) | Returns str, right-padded with pad to a length of len. If str is longer than len, the return value is shortened to len characters. In case of empty padstring, the return value is null. | |||||||||||||||
string | rtrim(string A) | Returns the string resulting from trimming spaces from the end(right hand side) of A. For example, rtrim(' foobar ') results in ' foobar'. | |||||||||||||||
array<array<string>> | sentences(string str,string lang,string locale) | Tokenizes a string of natural language text into words and sentences, where each sentence is broken at the appropriate sentence boundary and returned as an array of words. The 'lang' and 'locale' are optional arguments. For example, sentences('Hello there! How are you?') returns ( ("Hello", "there"), ("How", "are", "you") ). | |||||||||||||||
string | space(int n) | Returns a string of n spaces. | |||||||||||||||
array | split(string str, string pat) | Splits str around pat (pat is a regular expression). | |||||||||||||||
map<string,string> | str_to_map(text[, delimiter1, delimiter2]) | Splits text into key-value pairs using two delimiters. Delimiter1 separates text into K-V pairs, and Delimiter2 splits each K-V pair. Default delimiters are ',' for delimiter1 and ':' for delimiter2. | |||||||||||||||
string | substr(string|binary A,int start) substring(string|binary A,int start) | Returns the substring or slice of the byte array of A starting from start position till the end of string A. For example, substr('foobar', 4) results in 'bar'. | |||||||||||||||
string | substr(string|binary A,int start,int len) substring(string|binary A,int start,int len) | Returns the substring or slice of the byte array of A starting from start position with length len. For example, substr('foobar', 4, 1) results in 'b'. | |||||||||||||||
| string | substring_index(string A,string delim,int count) | Returns the substring from string A before count occurrences of the delimiter delim. If the count is positive, everything to the left of the final delimiter (counting from the left) is returned. If count is negative, everything to the right of the final delimiter (counting from the right) is returned. Substring_index performs a case-sensitive match when searching for delim. Example: substring_index('www.apache.org', '.', 2) = 'www.apache'. | GenericUDFSubstringIndex | ||||||||||||||
string | translate(string|char|varchar input,string|char|varchar from,string|char|varchar to) | Translates the input string by replacing the characters present in the | |||||||||||||||
string | trim(string A) | Returns the string resulting from trimming spaces from both ends of A. For example, trim(' foobar ') results in 'foobar' | |||||||||||||||
binary | unbase64(string str) | Converts the argument from a base 64string to BINARY. | |||||||||||||||
string | upper(string A) ucase(string A) | Returns the string resulting from converting all characters of A to upper case. For example, upper('fOoBaR') results in 'FOOBAR'. | |||||||||||||||
| string | initcap(string A) | Returns string, with the first letter of each word in uppercase, all other letters in lowercase. Words are delimited by whitespace. | GenericUDFInitCap | ||||||||||||||
| int | levenshtein(string A,string B) | Returns the Levenshtein distance between two strings. Example: levenshtein('kitten', 'sitting') results in 3. | GenericUDFLevenshtein | ||||||||||||||
| string | soundex(string A) | Returns the soundex code of the string. Example: soundex('Miller') results in M460. | GenericUDFSoundex |
Date Functions
In many analytical workloads Date is one of the most used built-in functions in Hive. The following list contains the supported built-in date functions in Hive.
...