Intro to Unix 2015
Intro to Unix 2015
Introduction to UNIX/Linux
Slide 1
About Me
• Worked at OSC for past two years as part of HPC Client
Services
• Graduated from The Ohio State University in Spring 14’
– Majored in Neuroscience, with Computational/Cognitive focus
– Minored in Computer Information and Engineering Sciences
• Grew up in Powell, OH; Attended Olentangy Liberty HS
Slide 2
The OH-TECH Consortium
Ohio Supercomputer Center provides high performance computing,
software, storage and support services for Ohio’s scientists, faculty, students,
businesses and their research partners.
OARnet connects Ohio’s universities, colleges, K-12, health care and state
and local governments to its high-speed fiber optic network backbone. OARnet
services include co-location, support desk, federated identity and virtualization.
Slide 3
Contents
• Introduction
• Useful Concepts
• Commands
• Files and Directories
• Linux Environment
• Scripting
• Selected Sources for Further Enlightenment
Slide 4
Introduction
• Learning outcomes of this course
• What is Linux?
• Why learn Linux?
• Software Uses
• Structure of Linux
Slide 5
Introduction – Learning outcomes of this course
• Introduce the basics of Linux
• To learn how to create/modify content in Linux
• Using Linux text editors to create documents
• Modifying file/directory permissions
• To learn how to write and run programs in Linux
• Provide hands-on practice
Slide 6
Introduction – What Is LINUX?
• Operating system and bundled application programs
• Available on virtually all machines in one form or another
• Derived from UNIX – Long history
• Adapted to new platforms
• Based on C programming language
Slide 7
Introduction – Why Learn Linux?
• Linux is FREE (open source)
• Linux is stable
• Linux systems are highly modifiable/customizable
• OSC clusters (along with many other HPC centers) use a
Linux distribution
• To use Glenn and Oakley effectively you need to know
some Linux
Slide 8
Introduction – Software Uses
• All Linux systems generally contain the following two types of
software:
• Operating system
– For the computer
– Liaison between computer and user
• Applications
– Basic functions
• Electronic filing
• Word processing
• Database maintenance (not in this workshop)
• Electronic mail and networking access (not in this workshop)
– Additional functions
• Programming
• Specialized IDEs/GUIs
Slide 9
Introduction – Structure of Linux
• Linux operating system software can be divided into two categories:
• Kernel
– CPU scheduling
– Memory management
– Process management
– In a nutshell: the kernel software handles internal communications that are
of no concern to the user of the system
• Shell
– Interacts between kernel and user, analogous to the “desktop” on other OS’s
– User invokes commands through shell
– Choices Shell
• BASH (default shell on OSC systems)
• Bourne
• Korn
• Cshell Kernel
• Tcshell
Slide 10
Introduction – Structure of Linux
Slide 11
Useful Concepts
• Standard Input – stdin
• Standard Output – stdout
• Linux is case sensitive
– The directory /nfs/12/kschooley
is different from /nfs/
12/Kschooley
– Linux commands – usually lower case
• ^ control key
– If ^ is part of a command, press the control key and the
second key simultaneously
• <Return> key
– Used to tell system we are at the end of command line
Slide 12
Commands
• Using the Online Manual
• Typical Command Structure
• Special Features
• First (and Last) Commands
• Exercise 1 – Log in
• Easy Commands
• Exercise 2 – Execute some simple commands
Slide 13
13
Commands – Using the Online Manual
Slide 14
Commands – Typical Command Structure
command –option argument
• command
– The name of an executable file
– Usually lower case
– What you want to do
• -option
– Sometimes not required
– Enhances/tailors the output of the command
– Often can be combined with one or more other options
• argument
– What command will act upon
– Often a command will have more than one argument
– Sometimes not required (implied)
• Commands are launched by pressing the <Return> key after typing
them
Slide 15
Commands – Special Features
• Can combine several commands on one line—separate
with semicolons
• Example:
- cd $HOME; mkdir newDir; cd newDir; emacs newfile&
- Can you guess what the result of executing this line of
commands will be?
Slide 16
Commands – Special Features
• Can create complex commands with redirection signs
- | (“pipe”)
- use the output of one command as the input to another
- Example:
- ls | grep my_dir
- cat myfile | grep neededinfo
Slide 17
Commands – Special Features
- >, >> (“output redirection”)
- redirects the output of the preceding command to a file
- > will overwrite the file if it exists, otherwise create a new file
- >> will append to the file if it exists, otherwise create a new
file
- Examples:
- ls > dir_list
- ls >> dir_list
Slide 18
Commands – Special Features
- <, << (“input redirection”)
- redirects input from a file to the preceding command
- < will simply redirect the input to be used as an argument
- << will redirect the input to be used in place of responses in
an interactive program
- Note: each “response” should be on a separate line
- Examples:
- mkdir < dirsList
- myInteractiveProgram << interactiveResponses
Slide 19
Commands – Special Features
• Can combine frequently used sequence of commands in
a file and run that file like a command (i.e., write a script)
– Syntax: ./executable_file
– We will discuss scripting in detail later on in this
presentation
• To send a process to the background:
– At execution time: add ‘&’ to the end of the command
– At runtime: bg %jobid
• To bring a backgrounded process to the foreground:
– fg %jobid
Slide 20
Commands – First (and Last) Commands
Slide 21
Connecting via OnDemand
Slide 22
Commands – Exercise 1
Slide 23
Commands – Easy Commands
• date
• cal [year]
• cal [month] [year]
• finger [username]
• who
• whoami
• echo
• env
Slide 24
Commands – Exercise 2
• Type the easy commands on the previous page, one at a
time, and look at the output.
• At the prompt type: man man
• Practice scrolling the man page, and exit
• Scroll slowly using down, up arrows
• Scroll down a page at a time using space bar (down),
“b” (up)
• Return to the beginning by hitting the “g” key, exit using the
“q” key
• Type man [command] for some of the commands on the
previous slide to view their documentation
Slide 25
Commands – Exercise 2
• Type some of the commands with the options from the
online manual pages. Can you combine some of the
options under one hyphen? (The options can’t be
combined if they contradict each other.)
• To answer a question you may have: you can use the
command echo and one of the commands, such as
date, in your shell programs so that you have an
automatic dating system for the resulting output. This
procedure is convenient for multiple runs of a shell script.
Check the command echo in the online manual pages.
Slide 26
Files and Directories
• Concepts • Manipulating Files and
Directories
• Typical Linux Directory
Structure • Permissions
• Where am I? • Exercise 4
• Getting Around the System • Searching and Information
Processing
• Exercise 3
• Exercise 5
• Naming Files and
Directories
Slide 27
Files and Directories – Concepts
• pathname
– Path through directory system to file
• Example: /usr/Workshop1/Subdirectory/file.name
– Absolute (full) pathname
• Shown when you type “pwd”
Slide 28
Files and Directories – Concepts
• File Descriptors
• Integer values assigned to open files by the kernel
• Standard Linux File Descriptors
• The following integers are reserved:
• 0 – “stdin”, or standard input stream
• 1 – “stdout”, or the standard output stream
• 2 – “stderr”, or the standard error stream
• Because stdin, stdout and stderr are recognized as files,
you can do many things with them that you can do with
ordinary files, as we will see later.
Slide 29
Files and Directories – Typical Linux Directory
Structure
Slide 30
Files and Directories – Where Am I?
Slide 31
Files and Directories – Getting Around the System
• Abbreviations for nearby directories
• . (dot)
– Current working directory
– Note: dot in LINUX overall has several uses
• .. (two dots)
– Directory above the one in which you are working (parent
directory)
• ~ (tilde)
– Home directory
Slide 32
Files and Directories – Getting Around the System
Slide 33
Files and Directories – Getting Around the System
Slide 34
Files and Directories – Exercise 3
Slide 35
Files and Directories – Naming Files and
Directories
• Avoid spaces; separate words with dots or
underscores
– my.file
Slide 36
Files and Directories – Manipulating Files and
Directories
• Create new directories using “mkdir” command
• Example: mkdir new_dir
• Example: mkdir –p parent_dir/child_dir
• Create new files using text editors, output redirection, or
the “touch” command
• Example: emacs new_file
• Example: ls -al > pwd_contents
• Example: touch filename
• Creates an empty file
Slide 37
Files and Directories – Manipulating Files and
Directories
• Another easy way to create a file:
• Type “cat > new_file”
• After this, just start typing…
• Press <Return> to start a new line
• Use ^d to return to the command prompt
• View the contents of your newly created file:
• cat new_file
• more new_file
• less new_file
Slide 38
Files and Directories – Manipulating Files and
Directories
• To move or rename a file or directory, use the “mv”
command
• Move a file to another directory:
• mv filename dirname
• Move a directory to another directory:
• mv src_dir target_dir
Slide 39
Files and Directories – Manipulating Files and
Directories
• Rename files/directories using the “mv” command
• Rename a file:
• mv filename newfilename
• Rename a directory:
• mv src_dir target_dir
• Question: How is this command different from the example
on the previous slide?
• Answer: In the previous case, target_dir already exists. In
this case, target_dir doesn’t exist yet.
Slide 40
Files and Directories – Manipulating Files and
Directories
• Copy files from one location to another using the “cp”
command
• Copy a file:
• cp filename target_dir
• cp filename1 filename2 target_dir
• Copy a directory:
• cp -R dirname target_dir
• cp -R dirname1 dirname2 target_dir
Slide 41
Files and Directories – Manipulating Files and
Directories
• Delete files/directories using the “rm” command
• Delete a file:
• rm filename
• Delete a directory:
• rm –r dirname
Slide 43
Files and Directories – Permissions
Slide 44
Files and Directories – “chmod” and “chown”
Slide 45
Files and Directories – “chmod” and “chown”
Slide 46
Files and Directories – Exercise 4
Slide 47
Files and Directories – Manipulating Files and
Directories
• Creating “links” between files using the “ln” command
• “hard link”
• Link points to the file’s metadata
• If you rename the original, the linked file will still point to the same
file. If you delete the original, the link goes away.
• Can only create hard links between files on the same filesystem
• “soft link”
• Link points to the actual file
• If you rename or delete the original file, the link remains but doesn’t
point to anything.
Slide 48
Files and Directories – Manipulating Files and
Directories
• Create a “hard link”:
• ln target link_name
• Create a “soft link”:
• ln -s target link_name
• Creating links between files can be useful for sharing
files with colleagues
Slide 49
Files and Directories – Exercise 5
• Issue the following commands to create hard and soft
links to a file in unix_training:
– ln unix_training/link_me hard_link
– ln -s unix_training/link_me soft_link
– Run “ls” to see the new links
– Run “readlink hard_link” and “readlink soft_link” to see
where the links point to
Slide 50
Files and Directories – Exercise 5
• Go into the unix_training directory, and run the
executable “link_me” to see its output
– cd unix_training; ./link_me
• Go back to the directory where your linked files are, and
run them as executables.
• Now, rename the file unix_training/link_me using the
“mv” command, and then try to run the linked files as
executables again:
– mv unix_training/link_me unix_training/moved_link
– ./soft_link; ./hard_link
Slide 51
Files and Directories – Searching and Information
Processing
• Regular expressions
– Used for pattern matching with strings of text
– Useful for searching for a particular pattern or string in
large files or large amounts of output
• Regular expressions, in general, have the following
components:
– An alphabet, or, a set of characters which define what you
are searching for
– The regular operations: repetition, union, and
concatenation
Slide 52
Files and Directories – Searching and Information
Processing
• The alphabet
– Usually represented as single characters or character sets
• The regular operations
– Repetition
• Usually denoted using the * (asterisk)
• Means, “repeat whatever directly preceded zero or more
times”
• Example:
– 0*1* (i.e., any number of 0s followed by any number of 1s)
Slide 53
Files and Directories – Searching and Information
Processing
– Union
• Set members in union surrounded by square brackets (“[ ]”)
• Means “or”
– Concatenation
• Usually implied (no symbol)
• Just put characters next to each other to imply concatenation
• Other commonly used symbols and “wildcards”:
– ? can be used to match a single character
Slide 54
Files and Directories – Searching and Information
Processing
• Regular expressions can be used with “ls” and “grep” to
search files with a particular pattern in their name
• ls uses all of the symbols mentioned on previous slides
– ls *.pdf
– ls [Hh]ello[Ww]orld
Slide 55
Files and Directories – Searching and Information
Processing
• Regular expressions can be used with the “grep”
command to search for a pattern in a text file or output
• “grep” recognizes most of the symbols on previous
slides, with a few minor differences/additions:
– Repetition operators:
• . (dot) – matches a single character
• * (asterisk) – matches zero or more of the preceding expression
• + (plus) – matches one or more of the preceding expression
• {n} – matches exactly n of the preceding expression
• {n,} – matches n or more of the preceding expression
• {n,m} – matches no less than n and no more than m of the
preceding expression
Slide 56
Files and Directories – Searching and Information
Processing
– Union operators:
• Set members in union surrounded by square brackets (“[ ]”)
• Negation symbol: ^ symbol used INSIDE square brackets
– Means, “match anything but the following characters”
• “OR” symbol: | (“pipe”) symbol used inside brackets or
parentheses to represent the union of expressions or
characters
– Concatenation operators:
• Default: no operator
• Parentheses can be used to group expressions
– Special symbol: implies concatenation
Slide 57
Files and Directories – Searching and Information
Processing
– Line anchors:
• ^expression – the ^ symbol used at the beginning of an
expression means, “match strings that begin with this
expression”
– Note the difference in meaning when the ^ symbol is used
outside square brackets
• expression$ – the $ symbol used at the end of an expression
means, “match strings that end with this expression”
– Escape character:
• \ - used to escape the meaning of special characters
(metacharacters and some whitespace characters)
• Note: Need to enclose the entire regular expression in
quotes to use with grep
Slide 58
Files and Directories – Searching and Information
Processing
• Grep syntax:
– grep expression file
– grep expression file1 file2
– expression contains grep regular expression operators, but the
filenames contain Linux regular expression operators
• Grep examples:
– grep hello unix_training/link_me
• Don’t need quotes around the pattern here, because it is a single
string with no whitespace
– ls unix_training | grep “_me$”
– grep “Found me at line [0-9]\{3\}” unix_training/grep
• In the above example, the curly braces must be escaped
Slide 59
Linux Environment
• Variables
• The “export” command
• Capturing Output
• Startup Files
• The “alias” command
• Exercise 6
• The Module System at
OSC
• Exercise 7
Slide 60
Linux Environment – Variables
• Similar to variables in other programming language
• Associate a descriptive variable name with a value
• Useful for storing information that is used often, but may
be too unwieldy to type manually every time it is used
• Make shell scripts more readable by eliminating the need
to type long pathnames
• Make it easy to run an executable file from anywhere on
the system
Slide 61
Linux Environment – Variables
• To use the value of a variable, precede its name with a $
• Example: echo $PATH
Slide 62
Linux Environment – Variables
• Global variables
• named in all capital letters
• Visible from the shell where created, as well as any “child” shells
• Local variables
• Named in all lowercase letters
Slide 63
Linux Environment – The “export” and “unset”
commands
• Use the “export” command to:
• Set the value of a new global variable
• export MYVAR=2
• export PATH=$PATH:$HOME/bin
Slide 64
Linux Environment – Capturing Output
• You can also capture the output of a command, and
store it in a variable
• Syntax for capturing output:
• `command arg`
• $(command arg)
• To store in a variable:
• export OUTPUT=`command arg`
• export OUTPUT=$(command arg)
Slide 65
Linux Environment – Startup Files
• Startup files are executable files that launch other scripts and/or
modify the environment at startup/login
• At OSC, there are system-level startup files that execute
automatically
• Users are able to create their own startup files
• .bash_profile – read first, contains environment variables by convention
Slide 66
Linux Environment – The “alias” command
• Use the “alias” command to create command aliases
• A command alias is similar to a variable, and is used for similar
reasons
• Associate a descriptive, custom command name for a command
you run often, but is too long to type every time
• Command aliases can be added to your startup files in order for
them to be automatically added to your environment
• Examples:
• Print a list of aliases: alias –p
Slide 67
Linux Environment – Exercise 6
• First, redirect the output of the “env” command to a file
• Use your favorite method for creating a new file, and
create a .bash_profile and test it out
• In your .bash_profile, set a few new variables and
command aliases
• Once you’re finished, log out and log back in for the
results to take effect
• Compare the results
• Hint: use “diff env_file `env`”
Slide 68
Linux Environment – The Module System at OSC
• Standardizes changes to the user’s environment per software
application
• All major software packages installed at OSC have an associated
module
• To see a list of all currently installed modules:
• On Oakley: module spider
Slide 69
Putting it all together: Scripting
• What Is a Script?
• Why Is Scripting Useful?
• Simple Shell Scripts
• Linux Text Editors
• Exercise 8
• PBS Batch Scripts
• The “qsub” command
• Exercise 9
Slide 70
Scripting – What Is A Script?
• A script is simply a sequence of commands written in a
file
• Each command is written exactly as you would type
them at the command prompt
• In other words, all of the syntax rules that apply when
running commands interactively apply to the syntax of
the commands you type into a script.
Slide 71
Scripting – Why Is Scripting Useful?
• Scripts serve much the same purpose that variables do
• They automate tasks you need to accomplish frequently,
but eliminate the need to type the entire sequence
manually every time
• Which is good, because some shell programs can be
quite large!
• Script writing is an essential skill for using OSC
resources effectively
Slide 72
Scripting – Linux Text Editors
• Various text editors exist for Linux
• At OSC, some of the more commonly used text editors:
• emacs
• vi
• gedit
• nano
• emacs has a GUI that is fairly straightforward to use
• To create a new file using the emacs GUI:
• emacs newfile&
Slide 73
Scripting – Exercise 8
• Create a simple shell script using emacs
• Set variables, run commands, be creative!
• Once the script is complete, run your program
• Note: you may need to set execute permissions for
yourself first.
Slide 74
Selected Sources for Further Enlightenment
• Internet (of course)
• Arthur, Lowell Jay, et al. UNIX Shell Programming, ISBN 0-471-16894-7
• Cameron, Debra. Learning GNU Emacs, ISBN 0-596-0064-9
• Dougherty, Dale. sed & awk, ISBN 1=56592-225-5
• Dyson, et al. UNIX Complete, ISBN 0-7821-2528-X
• Kernighan, Brian W., et al. The UNIX Programming Environment, ISBN
0-13-937681-X
• Kochan, Stephen G., et al. UNIX Shell Programming, ISBN
0-672-48448-X
• Lamb, Linda, et al. Learning the vi Editor, ISBN1-56592-426-6
• Lasser, John. Think UNIX, ISBN0-7897-2376-X
Slide 75
Selected Sources for Further Enlightenment
• Martin, Don, et al. UNIX Primer Plus, ISBN 1-57169-165-0
• Michael, Randall K. Mastering UNIX Shell Scripting, ISBN 0-471-21821-9
• Muster, John. UNIX Made Easy, ISBN 0-07-219314-X (my favorite)
• Ray, Deborah S., et al. Visual Quickstart Guide: UNIX, ISBN 0-201-35395-4
• Raymond, Eric S. The Art of UNIX Programming, ISBN 0-13-142901-9
• Reichard, Kevin, et al. UNIX in Plain English, ISBN 0-7645-7011-0
• Robbins, Arnold, et al. Classic Shell Scripting, ISBN 0-596-00595-4
• Robbins, Arnold. UNIX in a Nutshell, ISBN1-56592-427-4
• Robbins, Arnold. vi Editor, Pocket Reference, ISBN 1-56592-497-5
• Rosen, Kenneth, et al. The Complete Reference: UNIX, ISBN-13:
978-0-07-226336-7; ISBN-10: 0-07-226336-9
• Taylor, Dave. SAMS Teach Yourself UNIX, ISBN 0-672-32127-0
Slide 76