This article will explain how to use the uniq command in Linux to find or filter repeated lines in files and provide some usage examples.
The uniq command is a simple command which either outputs or omits repeated lines in the supplied input or file.
uniq Command Syntax
The syntax for the uniq command is as follows:
uniq OPTIONS INPUT OUTPUT
Note that:
- OPTIONS is a list of options from the below table
- INPUT should be the path to the file to be read
- *Standard input *can also be used
- OUTPUT should be the file in which the results of the uniq command are saved
- If not supplied, standard output will be used (by default, this prints the results to the console)
Options
The below table lists the commonly used options from the user manual:
| -c | Prefix lines by the number of occurrences |
| -d | Only print duplicate lines, one for each group |
| -D | Print all duplicate lines |
| -i | Ignore differences in case when comparing |
| -s | Avoid comparing the first N characters |
| -u | Only print unique lines |
You can view the full user manual for the uniq command by running:
man uniq
uniq Command Examples
Example File
The following examples will use the following example file as their input:
movies.txt
Dirty Harry Dirty Harry Total Recall Bride of Frankenstein Total Recall Demolition Man
Default Behaviour – Removing Repeated Lines
By default, the uniq command will remove repeated lines.
uniq movies.txt
Which will output:
Dirty Harry Total Recall Bride of Frankenstein Total Recall Demolition Man
Note that repeated lines are removed. Not duplicates. Only adjacent, identical lines are removed.
Showing Number of Repeats
The -c option will prepend each line with the number of times it was repeated:
uniq -c movies.txt
Which outputs:
2 Dirty Harry 1 Total Recall 1 Bride of Frankenstein 1 Total Recall 1 Demolition Man
Show Only Lines which are NOT Repeated
The -u option will show only lines which are not repeated:
uniq -u movies.txt
Which will output:
Total Recall Bride of Frankenstein Total Recall Demolition Man
Show Only Lines Which ARE Repeated
The -d option will show only repeated lines:
uniq -d movies.txt
Which will output:
Dirty Harry
Again – Note that repeated lines are included or excluded. Not duplicated lines. Only adjacent, identical lines are processed by the uniq command!
Saving the Result
Pass a filename as the final parameter to the command to save the results to that file:
uniq -d movies.txt duplicated.txt
Using Standard Redirection
As mentioned above, it is not necessary to supply input and output files to the uniq command.
Standard Redirection can direct the output of other commands and applications to uniq, which can then have its own output redirected to another program or file.
cat movies.txt | uniq > output.txt
In the above example, the cat command is used to read the example file – and the output of that is then piped to the unique command. The uniq command then has its output redirected to a file using the > operator.
Obviously, this doesn’t really provide any extra functionality to just providing the paths to input/output files. Still, it’s a clear example of how the output from commands can be piped/redirected to others.
