In most cases they’ve been provided by helpful users at Ubuntu Forums.
Remove duplicate lines from a file
This command is pretty useful in general, but is particularly handy for quickly removing duplicate entries from vocabulary lists.
awk ‘!x[$0]++’ file > output
In this command, file should be replaced with the input file, and output with the name of the new file which will have duplicate lines removed.
Remove all ASCII characters from a file
Removing non-ASCII characters is quite a common requirement, but this command does the opposite. It’s useful if you’ve got a file with a mix of Latin and CJK characters, and you want to leave only the CJK ones (e.g. delete everything except Chinese characters in a file).
sed -e ’s/[0-9a-zA-Z]//g’ -e ’s/[[:punct:]]//g’ -e ‘/^$/d’ file > output
Again, file is the input file and output is the file that will contain only CJK characters. Note that this isn’t perfect for isolating CJK characters - it will leave pretty much any unusual characters in the output file.
Get all lines with non-ASCII characters from a file
This command will go through a file and output any lines containing non-ASCII characters (such as Chinese, Japanese or Korean). This could be useful if you want to keep ASCII characters that appear on the same line as CJK ones, rather than completely removing everything except CJK (as the command above does).
grep -P “[x80-xFF]” *file* > *output*
Again, replace file with the input file, and output with the desired output file name.
Remove all lines containing X from a file
This command is useful for stripping out lines from a file that contain a specific term. I use it most often for removing lines marked with the word ‘simplified’ in vocab lists; it works well when you have tagged lists like that.
sed -e “/ text /d” file > output
Convert a .MO file to a readable .PO one (plain text)
A lot of software translations use MO and PO files to store a translated interface. The MO is the machine-readable one and is fairly useless if you want to get at the translations yourself. Use this command to convert an MO file into a plain text PO file:
msgunfmt *file* -o output
You’ll often want to get at the data in multiple .mo files at once, and this is easy to do by putting *.mo as the file in the command above. The command will still output the text into a single file.
Find the location of translation (.mo) files
To get hold of .mo files, you can either download language packs (e.g. language-pack-zh) and go through them with an archive browser, or you can search for the .mo files already in use in your installation. You can use the following command to do this:
dpkg -L language-pack
Replace language-pack with whatever language files you’re looking for, and the command will return the locations of all the .mo files for that language pack in your installation.
If you notice a mistake, or have another useful command to add, please share it in the comments.