Showing posts with label bash. Show all posts
Showing posts with label bash. Show all posts

Friday, September 28, 2007

Messing with arrays in bash

A couple of days ago, a problem was being discussed in #bash.

`How does one append to every element of an array?'

The usual answer is, run it in a loop. The better answer, however, is a one line parameter expansion thingie. Well, here's your answer:

$ array=( "${array[@]/%/foo}" )

That's it. One line of code to replace an entire for loop.

Here's a working example:

$ array=( foo bar baz )
$ echo "${array[@]}"
foo bar baz
$ array=( "${array[@]/%/foo}" )
$ echo "${array[@]}"
foofoo barfoo bazfoo

So how does it work?

Well, what we have here is actually the pattern matching and replacement operator from the Parameter Expansion facility of bash. The general syntax is as follows:

${parameter/pattern/string}

This way, the pattern to be matched can be replaced with the string in the value of the variable. When applied to an array index of either * or @, it performs the match and replace operation for every element of the array.

One curious feature of this operator is the use of %.

If the pattern to be matched starts with a %, it is matched at the end of the string. In our case, we've simply matched nothing at the end of the string and replaced it with foo. The result is that foo is appended to every element of the array. Cool eh?

Just be mindful of the double quotes in case your array elements have spaces or newlines in the values. :)



Once upon a time, I learnt that the expansion of "$*" or "${array[*]}" results in all the values separated with the first character of the value of $IFS. I always wondered where this feature could be used. Then all of sudden, I ended up using it twice in the space of two days.

The first problem was creating a few directories. Brace expansion was ideal. The values for directory names were coming from an array. Here's how I solved it.

(
DIRNAMES=( foo bar baz )
oIFS="$IFS"
IFS=","
eval mkdir -p /foo/{"${DIRNAMES[*]}"}
IFS="$oIFS"
)

I've put the entire thing in a subshell because I'm changing the value of IFS, which could be dangerous in a script. By using a subshell I'm making sure that rest of the script won't be affected. On top of that I'm saving the original value in $oIFS just to be sure. Yeah, paranoid. :)

eval is necessary because brace expansion happens before the parameter expansion takes place.

Anyway, "${DIRNAMES[*]}" actually expands to:

foo,bar,baz

So the whole command becomes:

$ mkdir -p /foo/{foo,bar,baz}

Nifty. :)

Here's another one.. I needed to feed some values from an array into a regex.

$ EXTENS=( txt c cpp h )
$ IFS="|"
$ awk "\$2 ~ /\.(${EXTENS[*])$/ { foo; }"

The expanded value becomes:

(txt|c|cpp|h)

:)

Monday, February 12, 2007

Processing procmail logs in bash

Well, the comments in the script should be self explanatory. It was awesome fun writing this! See if you `like' it ;)



#!/bin/bash
#
# Date: Monday 12th February 2007
#
# Processing procmail logs.
# It copies the original procmail log file to a
# temporary file called procmail after some grepping.
# After that, another temporary file procswapped is
# created and processed. Both files can be safely
# deleted afterwards.
#
# Only one drawback, it requires grep 2.5 or above.

# This is the input:
# From theobald@bar-plate.com Sun Feb 11 17:04:36 2007
# Subject: [SPAM] [+++++++++++++++] Re:
# Folder: /home/tnowak/Maildir/.Spam/new/_DzB.U6zzFB.dingo 13416
# From geuzzld@eriksbikeshop.com Sun Feb 11 17:30:40 2007
# Subject: [SPAM] [+++++++++++++] Still waiting
# Folder: /home/tnowak/Maildir/.Spam/new/_SFC.wS0zFB.dingo 30131
# From commercialtalk.com@esoleau.com Sun Feb 11 17:49:00 2007
# Subject: [SPAM] [+++++++++++] She will love you more than any other guy
# Folder: /home/tnowak/Maildir/.Spam/new/_5QC.8j0zFB.dingo 2309

# TomekN had run this much:

grep -B1 "\[SPAM\]" .procmail.log | grep -v "\--" > procmail

# Here's the procmail file at this stage:
#
# From trutawan056@yahoo.com Sat Jan 27 22:06:05 2007
# Subject: [SPAM] [+++++++++++++++] =?windows-874?B?odLDqNG0t9PhvLm608PYp8PRocnS
# From aw-confirm@ebay.com Sat Jan 27 22:39:56 2007
# Subject: [SPAM] [+++++++] You're a Silver PowerSeller Now!
# From 863kurtis@lightningdezignz.com.au Sun Jan 28 01:56:00 2007
# Subject: [SPAM] [+++] Fwd: Too busy to go back to school,{} but need a Un
# From trutawan055@yahoo.com Sun Jan 28 03:47:48 2007
# Subject: [SPAM] [+++++++++++++++++] =?windows-874?B?odLD46rp4rfDyNG+t+wgtdS0te

sed -ne '/^From/{
s/^/ /
h
n
s/^ *//
G
p
}' procmail > procswapped

# Here's the procswapped file at this stage:
#
# Subject: [SPAM] [+++++++++++++++] =?windows-874?B?odLDqNG0t9PhvLm608PYp8PRocnS
# From trutawan056@yahoo.com Sat Jan 27 22:06:05 2007
# Subject: [SPAM] [+++++++] You're a Silver PowerSeller Now!
# From aw-confirm@ebay.com Sat Jan 27 22:39:56 2007
# Subject: [SPAM] [+++] Fwd: Too busy to go back to school,{} but need a Un
# From 863kurtis@lightningdezignz.com.au Sun Jan 28 01:56:00 2007
# Subject: [SPAM] [+++++++++++++++++] =?windows-874?B?odLD46rp4rfDyNG+t+wgtdS0te
# From trutawan055@yahoo.com Sun Jan 28 03:47:48 2007

while read line
do
subarray[$i]="$line"
((i++))
done < <(egrep '^Subject' procswapped)

# I've read the Subject lines into an array called subarray
# Now I'll sort them according to the number of '+'s
# I've created a new array called plus, which has only the sorted
# '+' patters, including the [ at the beginning and the ] at the end.
#
# Use sort -r if you want the list to be reversed.

plus=($(for (( i=0;i<${#subarray[@]};i++ )); do echo "${subarray[$i]}" | egrep -o '\[\+*\]'; done | sort | uniq))

# Now the final work. The array plus is formatted into a form sed
# will understand as the address, by escaping the proper characters using
# guess what, sed!

for (( i=0;i<${#plus[@]};i++ ))
do
sed -n "/$(echo ${plus[$i]} | sed -n 's/\[\([^[]*\)\]/\\[\1\\]/p')/{
N
p
}" procswapped
done

# Here's the final output:
#
# Subject: [SPAM] [+++] Fwd: Too busy to go back to school,{} but need a Un
# From 863kurtis@lightningdezignz.com.au Sun Jan 28 01:56:00 2007
# Subject: [SPAM] [+++++++] You're a Silver PowerSeller Now!
# From aw-confirm@ebay.com Sat Jan 27 22:39:56 2007
# Subject: [SPAM] [+++++++++++++++] =?windows-874?B?odLDqNG0t9PhvLm608PYp8PRocnS
# From trutawan056@yahoo.com Sat Jan 27 22:06:05 2007
# Subject: [SPAM] [+++++++++++++++++] =?windows-874?B?odLD46rp4rfDyNG+t+wgtdS0te
# From trutawan055@yahoo.com Sun Jan 28 03:47:48 2007
#
# Sorted, based on the spam level indicated by the number of '+'s.

Sunday, January 21, 2007

Newlines and sed

Blogger gave me hell trying to post this, so I asked techno_freak to post it for me, on his blog; which he kindly agreed to :)

Thanks dude.

Here's the link to the post:

http://technofreakatchennai.wordpress.com/2007/01/19/newlines-and-sed/

Wednesday, November 08, 2006

A bash script to post on pastebin(s)

OK, ed-209 referred me to this script by zer0python, located here: http://www.alpadesign.com/shpost

I ended up adding features to it and in the process, making it ugly :-/. Anyway, here it is:


#!/bin/bash
# A Script that automates pasting to pastebin(s)..
# Thanks to ed-209 for this wonderful idea.. you can see his version
# which is located at http://www.alpadesign.com/shpost .. :>
#
# Author: floyd_n_milan
# Date: 08th November 2006
# Comments: Added features to the original script by zer0python
#
# Changelog:
# 08/11/06: Removed an unnecessary [ and added timeout to curl
#
# Usage: shpost.sh [-n nick] [-t type] [-s service] [-d description] \
# [-f source] [-h|--help|help]
######################################################################

function showusage
{
cat 1>&2 << EOF

nopaste: Automatic posting to pastebin(s).

Usage:
$0 [-n nick] [-t type] [-s service] [-d description] [-f source] [-h|--help|help]

Default nick is randomized.

type is one of the following:

"C89", "C", "C++", "C#"
"Java", "Pascal", "Perl"
"PHP", "PL/I", "Python"
"Ruby", "SQL", "VB"
"Plain Text"

BE SURE TO USE THE QUOTES FOR AT LEAST "Plain Text"

Default is Plain Text.

service can be one of the following:

rafb - http://rafb.net/post
sh - http://sh.nu/p/

Default is rafb.

Description must be quoted (""), if more than one word.

Default source is read from the keyboard. :-)

-h or --help or help shows this usage summery.

Mail comments, suggestions, bugs etc to
mrugeshkarnik@gmail.com
EOF
}

nick=""
lang=""
service=""
desc=""
input=""
url=""

if grep help <<<"$@"; then
showusage
exit 0
elif [[ ! $1 ]]; then
input="$(</dev/stdin)"
nick="shpostuser${$}"
lang="Plain Text"
service="rafb"
desc="shpost Post"
fi

while getopts ":n:t:s:d:f:h" opt; do
case $opt in
n )
nick="${OPTARG:=nopasteuser$$}"
;;

t )
lang="${OPTARG:="Plain Text"}"
;;

s )
service="${OPTARG:=rafb}"
;;

d )
desc="${OPTARG:="shpost Post"}"
;;

f )
input="$(<"${OPTARG:=/dev/stdin}")"
;;

h )
showusage
exit 0;;

? )
showusage
exit 64 #E_WRONGARGS (What's that?)
esac
done

input="${input:="$(</dev/stdin)"}"
nick="${nick:="shpostuser${$}"}"
lang="${lang:="Plain Text"}"
service="${service:="rafb"}"
desc="${desc:="shpost Post"}"

if [[ $service = rafb ]]; then
url=$(\
curl -i --connect-timeout 10\
-F "lang=$lang" \
-F "nick=$nick" \
-F "desc=$desc" \
-F "cvt_tabs=2" \
-F "text=$input" \
http://rafb.net/paste/paste.php 2>/dev/null | grep -i location)
echo "Your paste can be seen here: http://rafb.net${url:10}"
exit 0
elif [[ $service = sh ]]; then
curl --connect-timeout 10 -F "code=$input" -F "poster=$nick" http://sh.nu/p/
exit 0
fi

exit 0

Tuesday, November 07, 2006

bash Quoting

I thought I'd compile a list for bash's quoting rules for my own easy reference. I guess it might help others as well. Here goes then..

Escape Character (\)

Backslash (\) is used to remove the special meaning of the following character.
For example, \$ prints a $ instead of it being interpreted to signify a parameter.

An exception to the above statement is \. In the case of this sequence, bash looks for line continuation. Essentially, the character is removed completely.

Single Quotes ('')

Single quotes ('') are strong quotes. They bypass all the expansions. Everything inside single quotes is untouched.

You cannot have single quotes inside single quotes. Not even when backslash escaped. Use '\'' instead.

Double Quotes ("")

Double quotes allow parameter expansion, command substitution and arithmetic exansion. In short, all the expansions associated with $.

` is the archaic way of command substitution and is allowed inside double quotes.

\ inside double quotes is allowed only when used for \, `, $ and . In short, all the special characters which retain their special meaning inside double quotes.

Double quotes inside double quotes are allowed with used with a \. That is, \"

If history expansion is enabled, it'll be performed when ! is encountered inside double quotes. This can be bypassed with \!. The backslash preceding the ! is NOT removed.

Note: This can be annoying in your interactive shell when you get an error about history expansion when doing something like echo "Hello world!". The way to bypass this is to do "Hello world"\! or 'Hello world!' or Hello world\!

Examples:

Expression Value

$testvar hello
\$testvar $testvar
'$testvar' $testvar
"$testvar" hello
"'$testvar'" 'hello'
""$testvar"" hello
"\"$testvar\"" "hello"
~mrugesh /home/mrugesh
"~mrugesh" ~mrugesh
'~mrugesh' ~mrugesh

$' and $"

$'string' expands the string and the backslash escaped characters are replaced by the ANSI C standards. The expanded result is equivalent to being single quoted, as if $ is not present. Here are the characters expanded:

\a alert (bell)
\b backspace
\e an escape character
\f form feed
\n new line
\r carriage return
\t horizontal tab
\v vertical tab
\\ backslash
\' single quote
\nnn the eight-bit character whose value is the octal value nnn (one to three digits)
\xHH the eight-bit character whose value is the hexadecimal value HH (one or two hex digits)
\cx a control-x character

As an example:
$ echo -e 'Hello\nworld'
Hello
world

$ echo $'Hello\nworld'
Hello
world

$" translates the quoted string according to the current locale. For C and POSIX locales, the $ sign is ignored. If translated and replaced, the replacement is double quoted.

$* and $@

These two special variables are used in terms of positional parameters. They produce the same output when unquoted. When double quoted however, "$*" produces one word with all the positional parameters separated by the first IFS character; while "$@" produces different words, separated by spaces.

To elaborate, if "$*" is fed as an argument to a command, it is just one single argument ($# will show you 1). If "$@" is fed as an argument, it is several different arguments($# will show you N, where N equals the number of positional parameters provided.).

Most times, you'd want to use "$@".

When unquoted, the output of both is grounds for word splitting.

For example,

$ set -- "first argument" "second argument" "third argument"

$ for i in "$@"; do echo ">${i}<"; done
>first argument<
>second argument<
>third argument<

$ for i in "$*"; do echo ">${i}<"; done
>first argument second argument third argument<

$ for i in $@; do echo ">${i}<"; done
>first<
>argument<
>second<
>argument<
>third<
>argument<

$ for i in $*; do echo ">${i}<"; done
>first<
>argument<
>second<
>argument<
>third<
>argument<

This post is for a quick reference. Look here for a proper explanation: http://www.grymoire.com/Unix/Quote.html. This document is not restricted to just bash.

Friday, November 03, 2006

'bash'ing!

I love bash. I absolutely love it. It is fun. It is great fun! Take this simple script, posted by GreyCat in #bash for example:

x="hello world"; echo "$x" | read a b; echo "$a $b"; read a b <<< "$x"; echo "$a $b"

Quite simple, isn't it? Well, here's the output:


hello world

Hmm? OK. Let's try again.

hello world
hello world

What's going on?

OK. Now here's the beauty of this little script. It teaches so many fundamentals of bash.

We start off with a simple x="hello world". Next we have a pipeline. A pipe feeds the standard output of one command to the standard input of another command, as we know.

So here, we have the standard output of echo, which is 'hello world' (Well, that's written in English. In bash's terms, the output is equivalent to "hello" "world" I suppose..), being fed as the standard input of read. read stores the two words into two variables, a and b.

The pipeline finishes there and then we simply echo the values of a and b. The output, as we can see, is blank.

Now why exactly? Let's check.. Is the assignment of the variables correct? Yes. read takes the standard input by default, so a and b get assigned the values hello and world respectively. But the output of echo is still blank.

Why?

The answer is subshell. A pipeline spawns a different subshell for each process. So read a b operates in a different subshell.. different from echo "$a $b". As we know, subshells cannot propagate any information back to its parent shell. Hence, the values of a and b are blank, as they don't yet exist in the parent shell.

In the second case, we've used a form of here document - <<<. <<< expands the argument "$x" and feeds it to read a b's standard input. This time, we get a proper hello world output from echo "$a $b", because <<< doesn't spawn a subshell.

Now, if you run the same command line again, in the same session, you'll get this output:

hello world
hello world

How come?

Quite simple. The values of a and b are set by our previously run read a b <<< "$x". The values already exist in our shell, hence.

I suppose power users will find this information quite rudimentary. But, newbies would do well to understand the concepts involved. No matter how good I may become at bash, such small concepts will always be fascinating!