On a Linux machine I would like to traverse a folder hierarchy and get a list of all of the distinct file extensions within it.
What would be the best way to achieve this from a shell?
On a Linux machine I would like to traverse a folder hierarchy and get a list of all of the distinct file extensions within it.
What would be the best way to achieve this from a shell?
Try this (not sure if it s the best way, but it works):
find . -type f | perl -ne print $1 if m/.([^./]+)$/ | sort -u
It work as following:
No need for the pipe to sort
, awk can do it all:
find . -type f | awk -F. !a[$NF]++{print $NF}
Recursive version:
find . -type f | sed -e s/.*.// | sed -e s/.*/// | sort -u
If you want totals (how may times the extension was seen):
find . -type f | sed -e s/.*.// | sed -e s/.*/// | sort | uniq -c | sort -rn
Non-recursive (single folder):
for f in *.*; do printf "%s
" "${f##*.}"; done | sort -u
I ve based this upon this forum post, credit should go there.
My awk-less, sed-less, Perl-less, Python-less POSIX-compliant alternative:
find . -type f | rev | cut -d. -f1 | rev | tr [:upper:] [:lower:] | sort | uniq --count | sort -rn
The trick is that it reverses the line and cuts the extension at the beginning.
It also converts the extensions to lower case.
Example output:
3689 jpg
1036 png
610 mp4
90 webm
90 mkv
57 mov
12 avi
10 txt
3 zip
2 ogv
1 xcf
1 trashinfo
1 sh
1 m4v
1 jpeg
1 ini
1 gqv
1 gcs
1 dv
Powershell:
dir -recurse | select-object extension -unique
Thanks to http://kevin-berridge.blogspot.com/2007/11/windows-powershell.html
Adding my own variation to the mix. I think it s the simplest of the lot and can be useful when efficiency is not a big concern.
find . -type f | grep -oE .(w+)$ | sort -u
Find everythin with a dot and show only the suffix.
find . -type f -name "*.*" | awk -F. {print $NF} | sort -u
if you know all suffix have 3 characters then
find . -type f -name "*.???" | awk -F. {print $NF} | sort -u
or with sed shows all suffixes with one to four characters. Change {1,4} to the range of characters you are expecting in the suffix.
find . -type f | sed -n s/.*.(.{1,4})$/1/p | sort -u
I tried a bunch of the answers here, even the "best" answer. They all came up short of what I specifically was after. So besides the past 12 hours of sitting in regex code for multiple programs and reading and testing these answers this is what I came up with which works EXACTLY like I want.
find . -type f -name "*.*" | grep -o -E ".[^.]+$" | grep -o -E "[[:alpha:]]{2,16}" | awk {print tolower($0)} | sort -u
If you need a count of the file extensions then use the below code
find . -type f -name "*.*" | grep -o -E ".[^.]+$" | grep -o -E "[[:alpha:]]{2,16}" | awk {print tolower($0)} | sort | uniq -c | sort -rn
While these methods will take some time to complete and probably aren t the best ways to go about the problem, they work.
Update: Per @alpha_989 long file extensions will cause an issue. That s due to the original regex "[[:alpha:]]{3,6}". I have updated the answer to include the regex "[[:alpha:]]{2,16}". However anyone using this code should be aware that those numbers are the min and max of how long the extension is allowed for the final output. Anything outside that range will be split into multiple lines in the output.
Note: Original post did read "- Greps for file extensions between 3 and 6 characters (just adjust the numbers if they don t fit your need). This helps avoid cache files and system files (system file bit is to search jail)."
Idea: Could be used to find file extensions over a specific length via:
find . -type f -name "*.*" | grep -o -E ".[^.]+$" | grep -o -E "[[:alpha:]]{4,}" | awk {print tolower($0)} | sort -u
Where 4 is the file extensions length to include and then find also any extensions beyond that length.
In Python using generators for very large directories, including blank extensions, and getting the number of times each extension shows up:
import json
import collections
import itertools
import os
root = /home/andres
files = itertools.chain.from_iterable((
files for _,_,files in os.walk(root)
))
counter = collections.Counter(
(os.path.splitext(file_)[1] for file_ in files)
)
print json.dumps(counter, indent=2)
Since there s already another solution which uses Perl:
If you have Python installed you could also do (from the shell):
python -c "import os;e=set();[[e.add(os.path.splitext(f)[-1]) for f in fn]for _,_,fn in os.walk( /home )];print
.join(e)"
Another way:
find . -type f -name "*.*" -printf "%f
" | while IFS= read -r; do echo "${REPLY##*.}"; done | sort -u
You can drop the -name "*.*"
but this ensures we are dealing only with files that do have an extension of some sort.
The -printf
is find
s print, not bash. -printf "%f
"
prints only the filename, stripping the path (and adds a newline).
Then we use string substitution to remove up to the last dot using ${REPLY##*.}
.
Note that $REPLY
is simply read
s inbuilt variable. We could just as use our own in the form: while IFS= read -r file
, and here $file would be the variable.
None of the replies so far deal with filenames with newlines properly (except for ChristopheD s, which just came in as I was typing this). The following is not a shell one-liner, but works, and is reasonably fast.
import os, sys
def names(roots):
for root in roots:
for a, b, basenames in os.walk(root):
for basename in basenames:
yield basename
sufs = set(os.path.splitext(x)[1] for x in names(sys.argv[1:]))
for suf in sufs:
if suf:
print suf
I think the most simple & straightforward way is
for f in *.*; do echo "${f##*.}"; done | sort -u
It s modified on ChristopheD s 3rd way.
I don t think this one was mentioned yet:
find . -type f -exec sh -c echo "${0##*.}" {} ; | sort | uniq -c
The accepted answer uses REGEX and you cannot create an alias command with REGEX, you have to put it into a shell script, I m using Amazon Linux 2 and did the following:
I put the accepted answer code into a file using :
sudo vim find.sh
add this code:
find ./ -type f | perl -ne print $1 if m/.([^./]+)$/ | sort -u
save the file by typing: :wq!
sudo vim ~/.bash_profile
alias getext=". /path/to/your/find.sh"
:wq!
. ~/.bash_profile
you could also do this
find . -type f -name "*.php" -exec PATHTOAPP {} +
I ve found it simple and fast...
# find . -type f -exec basename {} ; | awk -F"." {print $NF} > /tmp/outfile.txt
# cat /tmp/outfile.txt | sort | uniq -c| sort -n > tmp/outfile_sorted.txt
For security reasons, it is desirable to check the integrity of code before execution, avoiding tampered software by an attacker. So, my question is How to sign executable code and run only trusted ...
How can I check the file encoding in a shell script? I need to know if a file is encoded in utf-8 or iso-8859-1. Thanks
i would like to know if there are any solution to do this. Does anyone? The big picture: I want to access data over the web, using my delphi thin clients. But i´would like to keep my server/service ...
Since cruise control is full of bugs that have wasted my entire week, I have decided the existing shell scripts I have are simpler and thus better. Here is what I have so far svn update /var/www/...
I ve just installed Zend Studio 7.0.2 on my Linux-Ubuntu 9.10 system. There were no problems during the installation but when I try to create a new project, the New Project form hangs when I click ...
I am running valgrind as follows:- /usr/local/bin/valgrind "process_name" After excecution its giving me following error ==21731== ==21731== Warning: Can t execute setuid/setgid executable: ==...
I was wondering if there were any good free graphics libraries for C that are easy to use? It s for plotting 2d and 3d graphs and then saving to a file. It s on a Linux system and there s no gnuplot ...
Is there anything other than DDD that will draw diagrams of my data structures like DDD does that runs on Linux? ddd is okay and runs, just kind of has an old klunky feeling to it, just wanted to ...