LPI 1 Article #6 Piping


In Linux, one program can manipulate the outputs from another program. This can be seen for instance if a text-filter can manipulate texts that are being produced by another program. This can be adequately done by the use of pipelines. What a pipe does (denoted by |) is that it redirects the outputs that a certain program produces into another program’s input. A good example can be a scenario where a file contains a lot of text and you are interested with a portion of the text in the file. Piping comes in handy by allowing you to direct the file to another program that will be able to extract the portion that you are interested in.
$ program1 | program2
For your information, pies can be used for an arbitrary number of sequence
$program1 | program2 | program3| program4 | program5..
The use of filters to process texts
The commands in this section does a lot of text manipulations from combining the files, transforming the available data in the files, displaying text, summarizing data and formatting text.
Commands that combine files
These are the commands that take two or more files and combines them into a single file. Examples of commands found in this category include: paste, cat and join.
i. cat command
cat combines many files end to end or rather an arbitrary number of files into one.
$ cat one.txt two.txt three.txt > file.txt
It is most likely that you will encounter cat being used to display the data in a file. That is the case too. It is used to show the data in a file to the standard output.
cat has got options
These include:
Showing number lines in the files that have been combined with -n or –number options
cat -n one.txt or cat --number one.txt : it numbers the lines in the file one.txt
To display the ends of lines
cat -E one.txt or cat one.txt --show-ends : a dollar sign will be seen at every end of a line in the file one.txt
ii.join command
The join command joins two files using a certain pattern in the files
An example is this
Let us say file1.txt contains the following contents:
34. Trump Jones
35 Philip Rice
37 Beatrice Gold
28 Stephen Cold
23 Felicious Fiph
And file2.txt contains the following files:
35 Present
37 Absent
28 Present
23 Absent

The command:
$ join file1.txt file2.txt
produces the following
34. Trump Jones Present
35 Philip Rice Present
37 Beatrice Gold Absent
28 Stephen Cold Present
23 Felicious Fiph Absent

The join command uses the first field to match the combination of files by default but this behaviour can be manipulated or altered using the options that the join command has.
The command
# join -1 2 -2 3 file1.txt file2.txt will join the using the second field of file1.txt and the third field of file2.txt.

iii. paste command
The paste command just combines the content of two files without using fields or anything else. It is reasonable to combine files that have the same number of lines to make it neater as shown below.
$ paste file1.txt file2.txt produces the following

34. Trump Jones 34 Present
35 Philip Rice 38 Present
37 Beatrice Gold 37 Absent
28 Stephen Cold 28 Present
23 Felicious Fiph 23 Absent

Creating command lines
Periods will come in your experience as a system administrator when you would wish or are required to do weird operations in your system such as deleting every file with a certain pattern or belonging to a particular user who is no longer part of the system user. One can picture looking for these files one by one deleting them and chills start rising because it can take quite a lot of time apart from the fact that it will be very boring. Relax admin, there are amazingly efficient ways that those tasks can be accomplished and we are going to learn about them here.
One of the most powerful tool you can use is the xargs command. Its syntax is as follows
# xargs [options] [command [initial-arguments]]
xargs does its work by running command once for each and every word it receives from the standard input.
Let us look at this scenario. You are required to delete every file in the current directory that belongs to user Mary. How will you go about it?
Well, by using many commands weaved together by xargs, the task can be accomplished very well. First, we are going to find the files that belong to Mary in teh current directory using the find command. After we find them, we are going to delete them using rm command.
# find ./ -user Mary | xargs -d "\n" rm
Th files belonging to Mary in he current directory depicted by ./ are being searched in the first part find ./ -user Mary. The output of find is piped to xargs which then adds those very values to rm command. The -d “\n” option for xargs tells it to use newlines only as delimiters. This is because files can have spaces in them and can create technicalities because xargs uses both spaces and newlines as delimiters of its items.
There is another way as well shown below but is less robust than xargs
# rm $(find ./ -user Mary)