Shell Utilities

Problem

  1. You want to compare 2 streams, with byte offset and line numbers?
  2. Pull off content with a byte offset?
  3. Display lines before or after a pattern?
  4. Send an email – upon job failure?


Solution

The shell is a powerful beast! Anything is possible … with a little know-how. 🙂 See the examples.



Example


  1. md5sum TARBALL | awk '{print $1;}' | cmp - TARBALL.md5
    # cmp good for comparing output with file contents, or diffing files. Outputs byte offset and line number. md5sum creates a hash (checksum) of file.
  2. dd bs=1 skip=106 if=infile
    # disk to disk copy, example outputs infile with offset of 106 bytes (block size of 1 byte, skip 106 bytes).
  3. Ever needed to display a line before or after a pattern. Yep sure you can use Perl, but this is a bit simplier to work out.

    See the code here

    See a run through screen shot here

    Usage – ppgrep #1 #2 pattern file – where #1 is the number of lines prior to the pattern (could be 0) and #2 is number of lines post pattern.

  4. Sending email upon UNIX cron failure




openssl many uses

Problem

  1. You want to generate a self-signed certificate?
  2. Encrypt a message with triple des
  3. View base 64 encoded data
  4. Generate and use random data
  5. Test ciphers
  6. Monitor certificate expiry dates


Solution

openssl to the rescue! 🙂 See examples.



Example


  1. openssl code to generate self-signed cert
  2. encrypt with tripledes
  3. openssl base64 [ -d ]
    # encrypted with base64.
  4. Useful fortransfering control/binary content – like this:

    dd bs=1 count=512 if=/dev/urandom | openssl base64

    Also useful for reading mail

    openssl base64 -d /var/spool/mqueue/qfk3GAlBMn018552

  5. Using openssl s_client to test ciphers supported, with automated script – then loop sites and email.
  6. Using openssl s_client to extract expiry dates, with automated script – then loop sites and email.


Reference

[tags]openSSL, dd, UNIX, SSL Certificates, PKI, Encryption, Generate Random Data, Unix Coding School[/tags]



uuencode send attachments

Problem

You want to send an attachment, from UNIX command line or from within a shell script



Solution

uuencode works a treat with Solaris. Not sure on RH Linux, etc.



Example



cat filename | uuencode wat_u_want_attatch_2b_called.ext | mail -s "this is the subject and here is wat u want attach 2b called" email_addr

Simple as that, it will send the filename as an attachment to a blank email, with your subject.

Here is a demo:

cat index.php | uuencode index.php | mail -s "test uuencode" junk@techie-blogs.com

To do multiple attachments, simply cat files – then uuencode them into a temp file. Like this:


cat file1.doc | uuencode attach1.doc > /tmp/send1.doc

Repeat this for say file2 into attach2 and send2. Then to send them all do this:

cat /tmp/send[0-9].doc | mail -s "your subj" youraddr





tar many uses

Problem

You want to make some backups with tar.



Solution

tar is very functional and has many uses.



Example


If you are lucky enough to have GNU tar (tar –version will return GNU if so), then you can compress and backup at the same time. 🙂

tar zcvf /tmp/filename.tgz .

This backups and zips the contents of the current directory, to filename.tgz in tmp directory.

If not – no drama. This will backup and pipe through content to gzip, which then zips it.

tar cvf - . | gzip --best > /tmp/filename.tgz

Recreate a directory:

mkdir new

tar -C old -cf - . | tar -C new -xvf

Restore over the network:

ssh remote_hostname "(tar -cf - remote_directory)" | tar -xvf -

For example:

ssh bree "(tar -cf - Perl_Bin)" | tar -xvf -



Reference

[tags]tar, backups, restores, Unix Coding School[/tags]



Port forwarding with SSH

Problem

You want to forward all traffic from a port on your box, to another host/port combination.



Solution

The command in the example will port forward, from the local box – any traffic sent to port 8081, will be sent to port 8080 on www.example.com:



Example


ssh -x -g -L 8081:www.example.com:8080 localhost

Don't forget, if you need to open LINUX (FC4) firewall, do this (as root):

vi /etc/sysconfig/iptables # add following line then save, exit
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 8081 -j ACCEPT
/etc/init.d/iptables restart





Email upon cron failure

Problem

You want to receive an email upon cron job failure.



Solution

The example code can be wrapped around a normal cronjob, to send an email if the cron fails.



Example


Add to beginning:

bash -c '( msg=`

Add to the end:

2>&1 > /tmp/logfail.log`; if [ ! -z "$msg" ] ; then echo "$msg" |

/bin/mail -s "`/bin/uname -n`:$LOGNAME:`/bin/date`: YOUR MESSAGE" YOUR_MAIL_LIST; fi )'

Demo

# # # # # bash -c '( msg=`YOUR COMMAND 2>&1 > /dev/null`; if [ ! -z "$msg" ] ; then echo "$msg" |/bin/mail -s "`/bin/uname -n`:$LOGNAME:`/bin/date`: YOUR MESSAGE" YOUR_MAIL_LIST; fi )'

Example of generating an email upon cron failure

Another example





Linux IPTables Open port range.

Problem

Had to find out how to allow a range with iptables recently, whilst setting up Samba and NFS.



Solution

Always take a backup first, then vi /etc/sysconfig/iptables. In the example I am allowing access to all ports between 600 and 699 – for udp and tcp traffic.



Example


# grep 600 /etc/sysconfig/iptables
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 600:699 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp -m state --state NEW -m udp --dport 600:699 -j ACCEPT

Then run /etc/init.d/iptables restart

You can test if this connection is now viable, by using my Perl port testing code under Network connections – Perl Coding School

Other gotchas with samba and nfs, was fiddling with selinux and ensuring portmap was running, etc. Happy to explain further, feel free to post a comment with your questions.



Reference

[tags]iptables, Unix Coding School[/tags]



Encrypt TripleDES

Problem

You want to encrypt some text, using the tripleDES Cipher.



Solution

Openssl is a beautiful command, that performs many functions. See example.



Example


Openssl tripleDES encrypt command:
openssl des3 -salt -in file_to_encrypt -pass pass:_your_password_

Security

Place contents to encrypt in the file reference by file_to_encrypt

Replace your_password with your secret

Openssl will output to stdout – so best to capture like this:
myvar=$(the openssl command)
Openssl tripleDES decrypt command:
openssl des3 -d -salt -in file_to_encrypt -pass pass:_your_password_
Put cipher text to decrypt in filename supplied to -in

Replace _your_password_ with the secret

Outputs to stdout
See the full demo attached here for a blow by blow:

[ How to encrypt with openssl and tripleDES ]



Reference

[tags]openssl, encryption, tripleDES, passwords[/tags]



Generate self-signed cert openssl

Problem

You want to generate a self-signed certificate, for use with a web server.

This will allow you to communicate with your web server over HTTPS, effectively encrypting your traffic.



Solution

The very first time – you need to set up your own CA (certifying authority). Do this one time only!

[ See a run through screen shot here ] [ See the code here ]

Then create a certificate request – which can be sent off to Verisign, etc or self-signed.



Example


To create a certificate request:

openssl req -new -days 730 -keyout keyna.pem -out keyna.pem

[ See a run through screen shot here ]

Self-sign the request – only do this if you are not sending off to a CA such as Verisign.

openssl ca -policy policy_anything -days 730 -out certna.pem -infiles keyna.pem

[ See a run through screen shot here ]

If you get 'unable to write random state', try this: look for command ssh-rand-helper usually under ssh install directories (or if ~/.ssh/.prng_seed exists – you can just link to it or copy it to $HOME).

ssh-rand-helper -b 1024 > $HOME/.prng_seed

To strip the password out, for restarts:
openssl rsa -in keyna.key -out keyna.key.unsecure



Reference

[tags]openssl, encryption, tripleDES, passwords, Unix Coding School[/tags]