Set up & how to use this site

Don't just read — type every command into your terminal as you go. The shell rewards practice, and muscle memory matters more than memorizing. Check off boxes as you finish each section; your progress saves in this browser.

The file system & where you are

The part of the operating system that manages your data is the file system. It organizes data into files (which hold information) and directories, or "folders" (which hold files or other directories).

When you open a shell you'll see a prompt — often a dollar sign — telling you the shell is waiting for input. Throughout this tutorial, lines starting with $ are commands you type; the rest is the output the shell prints back.

$ whoami
jovyan

When you type whoami and press Enter, the shell (1) finds a program called whoami, (2) runs it, (3) displays its output — here, your user ID — and (4) shows a fresh prompt, ready for the next command.

pwd — print working directory

Find out where you are with pwd. Your current working directory is the default directory the shell runs commands in unless you say otherwise.

$ pwd
/home/jovyan

At the very top of the file system is the root directory, which holds everything else. We write it as a single slash / — that's the leading slash in /home/jovyan.

Listing what's there with ls

Make sure you're in the data-shell directory, then list its contents with ls ("listing"):

$ ls
creatures  molecules           pizza.cfg
data       north-pacific-gyre  solar.pdf
Desktop    notes.txt           writing

ls prints the names of files and directories in alphabetical order, arranged in columns. Add the flag -F to mark directories with a trailing / so they're easy to tell apart:

$ ls -F
creatures/  molecules/           pizza.cfg
data/       north-pacific-gyre/  solar.pdf
Desktop/    notes.txt            writing/

Now you can see data-shell holds several sub-directories (trailing /) and a few plain files like notes.txt and solar.pdf.

Relative & absolute paths

You can point ls at something other than the current directory by giving it an argument — a path with no leading dash.

Relative path

List the contents of the data sub-directory:

$ ls -F data
amino-acids.txt  animal-counts/  animals.txt  elements/
morse.txt        pdb/            planets.txt  salmon.txt   sunspot.txt

data here is a relative path: it tells ls how to find something starting from where you are, not from the root of the file system. It has no leading slash.

Absolute path

Run it with a leading slash and you get a different answer — because /data is an absolute path measured from the root:

$ ls -F /data
ls: cannot access '/data': No such file or directory

You'll get a "No such file" warning here, because that directory doesn't exist at the root. The leading / tells the computer to start from the very top of the file system, so an absolute path always points to exactly one place no matter where you currently are.

To list this same data directory by its absolute path, you'd write:

$ ls -F /home/jovyan/data-shell/data/

That works regardless of your current pwd.

Two special shortcuts

Creating & deleting: mkdir, nano, rm

Now that you can explore, let's make things. Back in data-shell, create a new directory called thesis:

$ mkdir thesis

mkdir means "make directory" — it produces no output. Since thesis is a relative path, the new directory is created inside the current working directory. Confirm with ls -F, then check it's empty:

$ ls -F thesis

Editing a file with nano

Move into thesis and open the simple text editor Nano to create draft.txt:

$ cd thesis
$ nano draft.txt

Type a few lines, then press Ctrl+O to write (save) to disk, and Ctrl+X to quit back to the shell. (Unix docs often write ^O as shorthand for "Control-O.") Now ls shows your new file:

$ ls
draft.txt

Deleting with rm

Remove the file with rm ("remove"):

$ rm draft.txt

Run ls again and the output is empty — the file is gone.

Moving, renaming & copying: mv and cp

Re-create a file, then learn the two commands you'll use constantly to reorganize your work.

$ mkdir thesis
$ nano thesis/draft.txt   # type something, ^O to save, ^X to quit
$ ls thesis
draft.txt

mv — move or rename

mv ("move") takes a source and a destination. Give it a new name as the destination and it effectively renames the file:

$ mv thesis/draft.txt thesis/quotes.txt
$ ls thesis
quotes.txt

Give it a directory as the destination and it keeps the filename but puts the file somewhere new. The special name . (current directory) moves it right here:

$ mv thesis/quotes.txt .
$ ls thesis     # now empty

mv works on directories too — there's no separate mvdir command.

cp — copy

cp works just like mv, but leaves the original in place:

$ cp quotes.txt thesis/quotations.txt
$ ls quotes.txt thesis/quotations.txt
quotes.txt   thesis/quotations.txt

Both paths exist, proving you made a copy rather than moving the original.

Wildcards & counting with wc

The shell's real power is combining simple programs. Start in the molecules directory, which holds six .pdb files (Protein Data Bank format).

$ ls molecules
cubane.pdb    ethane.pdb    methane.pdb
octane.pdb    pentane.pdb   propane.pdb

wc — word count

cd into molecules and run wc ("word count"), which reports lines, words, and characters per file. The * in *.pdb is a wildcard that the shell expands into the full list of matching files:

$ cd molecules
$ wc *.pdb
  20  156 1158 cubane.pdb
  12   84  622 ethane.pdb
   9   57  422 methane.pdb
  30  246 1828 octane.pdb
  21  165 1226 pentane.pdb
  15  111  825 propane.pdb
 107  819 6081 total

Use -l for lines only, -w for words, -c for characters:

$ wc -l *.pdb
  20  cubane.pdb
  12  ethane.pdb
   9  methane.pdb
  30  octane.pdb
  21  pentane.pdb
  15  propane.pdb
 107  total

Wildcards

The shell expands wildcards into a list of filenames before the command runs, so wc and ls never see the * itself — only what it matched.

Redirecting output: > and cat

Which file is shortest? Easy with six files — but what about 6000? The first step is to save the counts to a file instead of the screen.

$ wc -l *.pdb > lengths

The > redirects the command's output into a file, creating it (or overwriting it) as needed. There's no screen output because everything went into lengths instead. Confirm it exists:

$ ls lengths
lengths

cat — show a file

cat ("concatenate") prints file contents to the screen:

$ cat lengths
  20  cubane.pdb
  12  ethane.pdb
   9  methane.pdb
  30  octane.pdb
  21  pentane.pdb
  15  propane.pdb
 107  total

sort & head

sort -n sorts numerically (without changing the file), and head -1 shows just the first line:

$ sort -n lengths > sorted-lengths
$ head -1 sorted-lengths
  9  methane.pdb

head -1 means "first line"; -20 would give the first 20. Since the list is sorted shortest-first, this is the file with the fewest lines. (< redirects the other way — feeding a file into a command's input, e.g. wc -l < mydata.dat.)

Pipes: joining commands with |

Those intermediate files (lengths, sorted-lengths) make things hard to follow. A pipe lets one command's output flow straight into the next — no temp files needed.

$ sort -n lengths | head -1
  9  methane.pdb

The vertical bar | sends the output of the command on its left as the input to the command on its right. Chain as many as you like — here, wc → sort → head with no intermediate files at all:

$ wc -l *.pdb | sort -n | head -1
  9  methane.pdb

A real example: sanity-checking data files

Imagine 1520 sample files that should each have 300 lines. Find the shortest five — a quick way to spot a truncated file:

$ wc -l *.txt | sort -n | head -5
 240 NENE02018B.txt
 300 NENE01729A.txt
 300 NENE01729B.txt
 300 NENE01736A.txt
 300 NENE01751A.txt

One file is 60 lines short — worth investigating. Swap head for tail -5 to check the longest files instead, and use a wildcard like ls *Z.txt to find flagged samples. Same handful of small tools, endlessly recombined.

Try it yourself

Work through these in the data-shell data. Predict the output first, then run it and see if you were right — that's how the commands stick.

$ mkdir recombine
$ mv proteins.dat recombine
$ cp recombine/proteins.dat ../proteins-saved.dat
$ ls

• Downloaded data-shell and cd'd into it
• Navigated with pwd, ls, and cd
• Created, renamed, copied, and deleted files
• Used a wildcard with wc
• Redirected output to a file with >
• Built a pipeline with |
• Completed exercises A–C

Command cheat sheet & key points

Key points to remember

• The file system stores files inside directories, which nest into a directory tree. / alone is the root.
• A relative path starts from where you are; an absolute path starts from the root.
• . = current directory, .. = parent, ~ = home.
• Most commands take flags beginning with - (like ls -F or wc -l).
• The shell has no trash bin — deletion is permanent.
• > redirects output to a file; | pipes output between commands. Combine small tools to do big things.
• Use Tab completion always — it's faster and prevents typos.

You've got the fundamentals. These same commands are what you'll use to drive Klone, run bioinformatics tools, and keep your projects organized. 🐚