Introducing MP3 Factory for 'de-borking' MP3s on Linux!

I had written a bunch of Bash scripts that i was using to process downloaded MP3 files with the objective of correcting problems and dumping all the junk i didn't want to save in my music files, such as cover art. I have now combined all these scripts, and much more, into what i'm calling MP3 Factory and i very much like the how things are panning out.

The script is in the alpha stage of development, but it is usable. It is also my first shell script, so not only am i learning a lot about some of the technical details MP3s, but i'm also learning the Bash syntax. You can check out my work on the MP3 Factory: De-borking MP3s on Linux page.

New guide: De-borking MP3s on Linux

I've been wanting to publish a guide for repairing and editing MP3s for a music collection on Linux for a long time. De-borking MP3s on Linux is nothing professional, but it's a starting point if you, like me, download a lot of music and want to repair errors in the files, normalize the volume and properly tag them. It's a work in progress and i'd appreciate any feedback.

Manjaro takes 1st place on Distrowatch

Yeah yeah, i'm late to the party, but i just now figured out that Manjaro Linux has captured the number one spot on Distrowatch, displacing Linux Mint. This happened some time within the last few months apparently.

If you're new to Linux, or want to remove the Windows virus, read on, else this could be about as interesting watching ice melt.

I find Manjaro's move to top dog status interesting considering it's based on Arch Linux which is notorious for being one of the more difficult flavors of Linux to install, configure and use. Manjaro however is specifically designed to be an easy-to-use Arch, complete with a capable graphical installer, package manager and software.

I started with Linux Mint a few years ago and i recommend it to anyone wanting to free themselves from the death grip of Microsoft. It's simple to install, feature rich and is probably one of the most polished and easiest to use distributions for newcomers. I became a bit frustrated with it because it's a 'point' release, meaning you have to re-install it when a new version hits the streets which, as i recall, was about every six months. Plus it's based on the LTS (long term support) version of Ubuntu which, in turn, is based on Debian. What that means is that the software in the Mint repository is often kinda old, forcing many to seek out lots of 'untrusted' PPAs, or figure out how to compile packages from source code.

Manjaro, on the other hand, is a rolling release, same as Arch, meaning you install it once and keep applying updates for ever more, in theory. Manjaro has it's own repository of considerable size, but one can also enable the AUR (Arch User Repository) which is also quite large (and can also get you in quite a bit of trouble).

Arch is pretty cutting edge and updates come fast and hard and can sometimes break the system. Manjaro receives a lot of updates too, many of which are quite large and affect somewhere around 100 packages at a shot, but the nice folks that work on the project alleviate some of the scariness by kicking the tires before turning packages loose.

I don't know that Manjaro is suitable for beginners, but it is definitely an attractive distribution. I've been using it for a few months and so far haven't had any major problems. If you're new to Linux and want to try it, just be sure to keep your data safe and learn first how to recover a busted system in the event something does explode, as has happened in the past.

A personal perspective: From Windows to Linux to Windows to Linux to...

The first personal computer i purchased was infected with the Windows 95 virus. I'm sure i hadn't even heard about Linux at that point and, while we're at it, let's get something straight right off the bat: Linux is not an operating system! Linux is a part of Linux-based operating systems known as the kernel. I'm a victim of this misunderstanding myself and, if there be no objections, i'm gonna go ahead and use the term 'Linux' as if i never knew the difference. Now where was i...

Anyone who has had the misfortune of running Windows 95 might agree that it was an unstable, bug-riddled piece of crap. The Windows 98 virus was much better, relatively speaking, but it was still full of annoyances. I have to say though that it was a pretty good gaming OS once you beat the thing into submission. The XP virus largely destroyed the gaming aspect of 98 when it was first released, but things improved a great deal as it matured. I used a heavily tweaked version of XP for many years before migrating to a heavily tweaked version Windows 7 somewhere around 2011. Us weathered Windows users were smart enough to skip Vista as though it never existed. By the way, did you know there are still people using 95 and 98 as their daily drivers apparently? I'm not sure what to think about that (actually i am, i just don't want to dump on them), but i wouldn't be surprised if they are safer with 9x than 10 these days since there's probably no one writing viruses for the damn things any longer.

Around the time i was using Windows 98 (or rather around the time it was using me, as Richard Stallman would say), i started to become interested in the inner workings of Windows, computer security and digital privacy. The exceedingly bright chap i am (0.003% of the time), i quickly deduced that running a closed-source, proprietary operating system from a company well-worth hating was probably not conducive to maintaining an acceptable level of control over my digital do-dads. This revelation was further reinforced upon discovering the forum post, 'Microsoft's Really Hidden Files' by "The Riddler" on fuckmicrosoft.com. I believe that domain was eventually hijacked by Microsoft at one point which i think is quite fitting, don't you? I think they should 'ing' it to 'fuckingmicrosoft.com' and use that as their official domain. Anyway, my reading of that article eventually led to my being put in contact with a cop who, as i recall, had something to do with computers or computer forensics or was the computer geek at work or maybe he knew what a computer was or something. It was a long time ago. I asked him several (and apparently one too many) questions about what kind of personal data Windows stores and how it stores it and how that data can be used by LE (that's "Law Enforcement" for you nooblets) to screw over those that paid for an OS which is designed to spy on them. Having thought i'd established an acceptable level of trust with the guy after several emails over the course of several days/weeks (it's called "social engineering" nowadays), i finally uncorked the question that was burning a hole in my curious little bean: Can LE access a Windows computer remotely without first installing a RAT (Remote Access Tool - think "Trojan")? His non-answer answer was, "What do you think?". At that point he apparently thought i didn't need to ask any more questions.

I don't think it was long after that when i started playing with Linux... or was it the other way around? I recall booting Mandrake, Red Hat, SUSE, Debian and maybe one or two others over the years and every attempt i made to familiarize myself with it resulted in tragedy followed by regret followed my prompt retreat back to my safe space: Windows. Part of my Linux problem was that it worked quite differently and so i simply wasn't comfortable with it. Another problem was that every distribution had major bugs. Lots of major bugs. And they still do. All of them. I found Linux to be very fragile compared to Windows. I also found it to be too limiting, which i'm sure will shock some Linux fanboys. Often there weren't any good software counterparts to Windows stuff and many hardware drivers were garbage. If a driver worked at all, that's pretty much all it did. Most of the granular settings for coaxing the most out of your uber-expensive octa-core giga-capable quantum-tech hardware simply didn't exist. The lack of stability was the worst though. I remember booting a very mainstream distribution and changing some benign setting for whatever and *poof*, it never booted again. On another occasion i was playing around with another mainstream distribution when my drunk neighbor chainsawed my electric pole because he needed firewood (lie) which caused the power to go out (semi-fact). This was before i knew what a backup power supply was. That OS never booted again either. What the hell??? I rarely had these kinds of fundamental problems with Windows 98, much less with XP which was a full 3.307 orders of magnitude more robust than any Linux distro i had tried.

Though i was fine with Windows from a usability angle, the fact that it was propitiatory, and the associated risks to my privacy that goes with that, never left my mind and so i decided, come hell or high water, Windows 7 would be my last Microsoft OS, period! Then came Windows 10 but once again we savvy Microsoft-hating Windows users were smart enough to block the so-called "upgrade" that the monopolistic retards in Redmond wanted to infect us with. The in-your-face assault to personal privacy that 10 presented was/is unbelievable. Even though i wasn't running 10, I was so disillusioned with Microsoft at that point that i dumped Windows, switched to Linux and deleted all my heavily customized Windows installation images and license keys in order to make returning to that pit of hell as difficult and expensive as possible should i ever suffer a relapse in ethics. I was going to run Linux whether i liked it or not and that was that! This video of a guy ranting about Windows 10 kind of sums up one of my many beefs with Windows:

Linux has come a long way since i last went distro-hopping well over a decade ago, but i have to say that some of the same problems exist still, though to a lessor degree in general. And although the drivers cover a much wider array of hardware, they can still be problematic and wimpy. For example there's nothing like the good ol' kX Project sound card drivers which transformed some old Sound Blaster cards into a million dollar sound studio, free of charge. Nowhere was i able to find a driver for my laptop camera either, not that installing one would help much since i have tape over the thing. Stability is still an issue as well, though to a lessor degree and it appears Linux is more tolerant of hard resets. I'm currently running Linux MintManjaro Linux, but when i was running Mint, and even after a couple of major version releases and plenty of updates, on occasion my computer would still boot or resume from standby to a black screen where the log-on screen was supposed to be. Really??? This is one of those most basic issues that should simply not exist.

And the software repositories! Why 'repositories' and not 'repository'? Why does nearly every major distribution of Linux seem to need its own software repository? To me this is a colossal waste of time and resources, including the server infrastructure required to host all this. Why do package maintainers have to make changes to a program to get it to run on different distributions? Or another desktop environment? Is this not as ass-backwards as i think it is? This can be really frustrating for end users since the software package they want may not be available for the distribution they like, or perhaps not for the particular desktop environment they chose. Oh, yeah, let's not forget the graphical environments; the Linux desktops. Linux is all about choice, so they say, but do we really need 20+ buggy desktop environments to choose from? I'd gladly sacrifice most of those for just a couple of bug-less desktops and i'd bet 99.999% of the entire Linux user base would do the same.

And why are there so many different distributions? There are somewhere around 300 different flavors of Linux at the moment, every one likely requiring significant resources in terms of developers, project managers, funding and infrastructure. Who the hell needs this many choices? And more importantly, at what cost do all these choices come? How many of these distributions are a result of ego battles? How many are actually fundamentally different as opposed to simply having a different logo or set of icons or something trivial? Imagine how the development pace and quality of Linux might skyrocket if the talented people working with some of the more obscure distributions contributed to the more mainstream ones that the vast majority of us end users actually use. Here's some more of the "choices" you get with Linux:

  • you get to choose what distro you want and, in doing so, the average user has little choice but to use whatever software is in the repository for that distro
  • you get to choose the desktop you want to and, in doing so, you further neuter the selection of available software or end up installing mega-massive dependency packages you don't want
  • you get to choose that you want more security holes than Windows if this guy is right
  • you get to choose that you want truck loads more bugs than Windows
  • if you decide to run a Long Term Support edition (Linux Mint for example), you get to choose to run often outdated, buggy and feature deprived software because that's pretty much all that will be available in your assigned repository
  • you get to choose to spend hours scouring the web for solutions to trivial problems that shouldn't exist and then more time staring at a terminal running commands you don't understand but pray will work and often don't
  • you get to choose to have this window look like this and that window look like that because you hate consistency - think that setting a window or mouse cursor theme is system wide? HA!
  • you get to choose to run an OS that just feels a lot less robust than Windows

So why did i choose Linux if i have so many gripes, you ask? Because i had no choice. I wasn't going to continue to run an operating system that i knew was spying on me, constantly sending who knows what to who knows where for who knows why. Even with all the registry hacks and all the hosts file redirects and all the firewall rules and all the 3rd party cleaners and anti-spying tools, etc., you can never ever trust a closed source, proprietary operating system, period, and that's not an option for us privacy lovers.

And as far as the stability i enjoyed with Windows, i very well may be an exception to the rule as there are tons and tons of horror stories involving botched Microsoft updates that caused massive problems for many users and businesses. I attribute my avoidance of such problems to my software selectivity which allowed me to avoid the garbage and malware, as well as selective patch installs, and possibly a dose of good luck.

The following presentation, 'Windows is AWSOME!' by Bryan Lunduke, which he gave at a Linux conference by the way, will provide some key answers as to why i made the switch. It's a fun watch...

For the average user Linux may be as good as Windows from a usability perspective and it's not authored by a f'n evil company. It's not proprietary. It's code isn't hidden. It doesn't have back doors built in. Its licenses aren't restrictive. It isn't designed to spy on its users and make them the product. Its objectives aren't based on greed and control. In a way, it's not what Linux is, it's what it isn't that makes it attractive to people like myself who care privacy and what organizations, values and standards we want to support. Some of us care about what kind of world we want to live in and what kind of world we leave for our children. There is more to the free software ethic than "free" software. It's a life choice.

Although my path to Linux hasn't been free of landmines, it has been worth the journey. It's very comforting to know that i can trust my operating system to not engage in nefarious behavior behind my back. It's nice to not have to create piles of firewall rules in an attempt to keep it and its software from phoning home. It's nice to not have to evaluate every single update, wondering if Microsoft will try yet again to push some despicable unwanted garbage down my throat. It's nice to not have to deal with anti-virus software, which provides a false sense of security anyway. It's comforting to know that Linux doesn't lend itself well to viruses and unethical software developers or companies. And as for the problem of multiple software repositories, at least one can trust the software in those repositories and they do present a convenient one-stop-shop for most everything you need.

Unfortunately your computer runs more than one 'operating system' and so installing Linux alone isn't a complete solution for us privacy and security geeks. Various motherboard firmware performs various tasks before the system is handed over to the user-facing OS and some of this stuff is a privacy and security nightmare. Perhaps the most egregious example of this is the Unified Extensible Firmware Interface (UEFI) which is replacing the traditional Basic Input/Output System (BIOS). You can read more about this massive assault to your privacy and computer security in the articles 1.4 UEFI... The Microsoft Kill Switch and The Fight for a Secure Linux BIOS. You gotta start somewhere though and Linux is a massive step up in terms or privacy and security.

If you decide to give Linux a spin, i might recommend Linux Mint because it's currently one of the more popular and polished Linux distributions that's oriented toward beginners. I think Windows users will find its interface fairly familiar and it comes with a lot of decent software as well as most/all of the hardware drivers needed to get stuff up and running. To try Mint, or most other popular distros, you just have to write an image to a USB memory stick and boot it. You can try it for as long as you like without affecting your Windows virus and if you later decide to install it, you can do so right from an icon on the desktop. Everything you need to know should be in the installation guide.

If you ever get bored with Mint and begin to feel more adventurous, have a look at Manjaro which is quickly becoming a stable, user friendly Arch-based OS. It's also a rolling release meaning that, unlike most distributions, you don't have to reinstall it when there are major updates.

In the end, Linux is miles from perfect, but it runs on virtually any hardware you can imagine, new and old, and it provides a degree of trust that is nonexistent with any proprietary computer operating system.

Good luck. And if there's anything i can do to help, leave a comment.

MP3 Factory: 'De-borking' MP3s on Linux

Note that MP3 Factory is under development and documentation is not complete. Use at your own risk. Lastly, the options set in the configuration file may not be set to their default values.

intro

Yes, "de-borking" is a word you grammar Nazi! It's right here in the dictionary :)

Apparently i'm an oddball because my MP3s usually contain only the artist, title and genre tags (and i only use 3 or 4 of the latter), along with the other required stuff and nothing else, except for the actual audio of course. I'm not interested in embedding cover art, lyrics, soupie recipes or whatever else people are sticking in their music these days. I like to aim for minimal, compact, error-free files that meet my (medium-high) quality standard and that's what this is all about. If you're a super serious audiophile that spends days deciding which DAC will provide the truest sound (check out this one), this might not be for you, but if you just like to listen to good sounding music while you do whatever it is you do, read on.

In an ideal world of unlimited availability and smartphones with unlimited storage one may never mess with MP3s, preferring instead one of the lossless formats such as FLAC, however some songs are difficult or impossible to find in such a format and storage space is still pretty limited on our fondleslabs, so making efficient use of that space still seems like a good idea. Also the MP3 format generally provides a transparent listening experience as long as the encoding was performed with quality in mind, transparent meaning that it is unlikely that anyone with good hearing, in a quiet environment, with good equipment, would be able to differentiate between a high quality MP3 and the original lossless recording.

Typically i acquire specific songs from various artists rather than entire albums because i've never run across an album for which i liked every song. The problem with this disjointed approach is that downloading songs willy-nilly usually results in problems regarding tags, header errors, too much silence, nonuniform volume, etc.. Most maddening is when some aspiring government employee thinks that upsampling a 64 kbit/s MP3 to 320 kbit/s is somehow a good idea. With the exception of the latter, such problems are generally easy to fix using a few software tools.

Enter MP3 Factory, a Bash shell script which is essentially a wrapper that handles various tools and does so without relying on crap i don't like relying on, such as Java, Wine and Mono.

MP3 Factory, or MP3F if you rather, is intended primarily for transcoding (converting) downloaded music files to MP3s and/or performing various operations on them before adding them to your collection. Several of the functions that MP3F is capable of are applicable to formats other than MP3 however. With the exception of transcoding non-MP3's to MP3, all of the tasks which it performs are lossless, meaning that there is no loss of sound quality.

MP3 Factory is not a complete solution for processing music files. For example only rudimentary tagging functions are available (for greater tagging flexibility you might consider a good graphical program like puddletag), nor is it a complete replacement for comprehensive MP3 analysis and repair software like the venerable MP3 Diags which can address certain issues with MP3 files that MP3 Factory cannot.

MP3 Factory was developed and tested on Manjaro Linux or, as i like to call it, 'Arch for dummies'. The script should run on any flavor of Linux that includes a Bash compatible shell and for which the dependent packages are available.

highlights

  • Batch processing
  • Convert to MP3
  • Convert tags to file names and file names to tags
  • Repair and optimize files
  • Integrity checking, including LAME CRC testing
  • Generate frequency spectrographs for graphic analysis
  • Trim excess silence interactively
  • Normalize gain (volume) with ReplayGain 2.0 data
  • Process files other than MP3
  • Play a video of a music file
  • Output clean, error-free, compact MP3 files
  • Create multiple configuration files
  • The code is ShellCheck and Shellharden compliant, however that certainly doesn't mean it's bullet-proof so if you find a bug, let me know.

overview

The MP3 Factory script runs in a terminal, but don't let that scare you half to death if you're not a terminal freak... get it? There are plenty of prompts to guide you along and it presents a unified interface for running several software tools which are available for Linux-based operating systems (because they suck slightly less than Windoze).

Music files can be batch processed and automatically retained or discarded according to parameters you define, or the operation can be interrupted after each file is processed so you can decide whether a file is worth keeping.

Following are the primary functions offered by MP3F:

1) Tag To File Name     8) Write A Tag        15) Repairs (!)
2) Integrity Check 1    9) Auto-Tag           16) Add Log Entry
3) Spectral Analysis   10) Optimize           17) View Log
4) Clean File Name     11) Integrity Check 2  18) Edit Config
5) Convert To MP3      12) Play Audio File    19) Load Config
6) Clean Metadata      13) Collection Stats   20) Documentation
7) File Name To Tag    14) Trim Silence (!)   21) Quit
  1. Tag To File Name : Rename a file using ID3 metadata (tags).
  2. Integrity Check 1: Run various Integrity Checks on the files and discard those which do not pass.
  3. Spectral Analysis : Create a spectrograph image of the file to check for various issues.
  4. Clean File Name : Removes unwanted characters from file names.
  5. Convert To MP3 : Convert a non-MP3 to an MP3.
  6. Clean Metadata : By default this task removes all obsolete and ID3 metadata and then writes back only the requested ID3 metadata in the desired format.
  7. File Name To Tag : Write ID3 tags derived from the file name, such as artist and song title.
  8. Write A Tag : Write user defined ID3 tags to the file.
  9. Auto-Tag : Writes multiple, predefined ID3 tags to the file.
  10. Optimize : By default this task reverses changes made by mp3gain when possible, writes ReplayGain tags and removes all ID3 padding in order to minimize the file size. ReplayGain information is written using loudgain, a byproduct of which is the removal of all ID3v1, ID3v1.1, ID3v2.2 and APEv2 tags.
  11. Integrity Check 2 : This second Integrity Check performs all of the tests the first one does, plus additional metadata checks.
  12. Play Audio File : Play a music file using a choice of players.
  13. Collection Statistics : Provides various quality and conformity statistics for all files in a folder.
  14. Trim Silence (!) : Remove excess silence from the beginning and end of the file. See the 'trim silence' section below for important notes regarding this task.
  15. Repair (!) : Repair a damaged file (damaged audio frames cannot be repaired). See the 'repair' section below for important notes regarding this task.
  16. Add Log Entry : Add a user defined comment to the session.log file.
  17. View Log : View the current log file.
  18. Edit Config 1 : Edit the script options in the default.conf file.
  19. Load Config : Load a different configuration file.
  20. Documentation : View various included and on-line manuals and documentation.
  21. Quit : Perform housekeeping and exit.

Footnotes:

(1) Be extra careful to not mess up the syntax when editing the configuration file(s). Not even kidding. These files are sourced from the main Bash script and thus are essentially a part thereof.

installation

Prior to using MP3 Factory you should check that its dependencies are available for your distribution. The script will notify you of any which are missing.

Primary dependencies...

  • python-eyed3 : A "tool for working with audio files, specifically MP3 files containing ID3 metadata (i.e. song info)."
  • apetag : A utility for reading and writing APEv2 tags, used only to remove APE tags.
  • exiftool : "... a platform-independent Perl library plus a command-line application for reading, writing and editing meta information in a wide variety of files." exiftool is used by MP3F only to read metadata since it cannot write to the MP3 format.
  • eyeD3 : "... a Python tool for working with audio files, specifically mp3 files containing ID3 metadata"
  • ffmpeg : "A complete, cross-platform solution to record, convert and stream audio and video."
  • flac : For processing FLAC files.
  • loudgain : "A versatile ReplayGain 2.0 loudness normalizer".
  • mp3gain : "Analyzes and losslessly adjusts mp3 files to a specified target volume". mp3gain is used primarily to un-do the damage it caused to a file, though this is not always possible.
  • mp3guessenc : "A small utility for analysis of audio mpeg files and detection of the encoder used".
  • mp3info : Used to obtain various information about an MP3.
  • mp3splt : A "utility to split mp3, ogg vorbis and native FLAC files selecting a begin and an end time position, without decoding". mp3splt (yes, that's how it's spelled) is used to trim excess silence from an MP3 or FLAC file.
  • mp3val : A "tool for checking MPEG audio files' integrity".
  • rmlyrics3 : "A Lyrics3 tag remover for MP3 files".

Optional dependencies...

  • notify-send : Provided by the libnotify package, it is used to send desktop notifications to notify you when a batch process has completed.
  • shellcheck : "Finds bugs in your shell scripts". It is strongly suggested to install this package so that the configuration files for MP3 Factory can be checked.

After downloading the MP3 Factory archive, extract it somewhere where you have read, write and execute permissions, such as in your home directory. The mp3factory.sh script will be contained in a folder named 'mp3actory'. I know you never anticipated such stunning creativity, but there it is! Don't forget to make sure it's executable ( $ chmod +x mp3factory.sh ).

MP3 Factory will create the following directories and files within the 'MP3 Factory' folder:

/.temp    (holds a copy of each file being processed)
/backup   (copies of source files converted to MP3)
/config   (configuration files)
/holding  (for holding files you want to process)
/junk     (discarded files)
/logs     (take a guess)
/spectro  (spectrograph images of music files)
/working  (files which are to be processed)
junk.txt  (file names of discarded files)

Other files included in the archive are:

help.txt       (internal help file)
license.txt    (GPL license)
mp3factory.sh  (MP3 Factory shell script)

The first thing you'll want to do is edit the configuration options in the /config/default.conf file using a capable code editor. Better yet, make a copy of it and edit the copy (give the copy a descriptive name and a .conf extension). If you're running the KDE desktop then Kate is a nice choice for a code editor, just don't use something like LibreOffice Writer or whatever. If your editor supports syntax highlighting, set it to 'bash' or 'ini' to help avoid errors. Carefully review each option and their descriptions and make the necessary changes. Be careful to not make any syntax errors.

If you need help with the options for a dependency program, such as for ffmpeg, try one of the following in your terminal. Note that the name of the program is typically case sensitive and usually all lower case, eyeD3 being the red-headed stepchild of the lot.

man <program name>
<program name> --help
<program name -h

configuration

You can create multiple configuration files in the /config folder in which case MP3F will ask which one you want to load when you run the script. The suggested way to do this is to copy the default.conf file to a new file and edit the new file. Be sure to give it a descriptive name and a '.conf' extension.

There are 3 types of option/value pairs in the configuration file and in all cases there can be no space between the option name, the equals character, and the value, and all 3 must be present. Do not alter any of the option names. Following are some examples of properly and poorly formatted options:

Good : optionName='value'
Bad  : optionName="value"
Bad  : optionName = 'value'
Bad  : optionName'value'
Bad  : optionName=

The types of options in the file are strings, integers and arrays. For a string the entire value must be surrounded by single quotes:

Good : stringName='some string'
Bad  : stringName=some string

For an integer the value must be a whole number and must not be quoted:

Good : integerName=1
Bad  : integerName=1.2
Bad  : integerName='1'

Arrays must be enclosed in parenthesis and each element must be enclosed in
single quotes and separated with a space (there may exceptions where some
values may be double quoted):

Good : arrayName=( 'value1' 'value2' '...' )
Bad  : arrayName=('value1''value2''...')
Bad  : arrayName='value1' 'value2' '...'

All lines beginning with a ' # ' character are comments and are ignored by the
shell. You may add, edit or remove comments, however do not remove or alter the " # default value: ... " lines as they are used to reset all the values if you run mp3factory.sh with the -r option.

You should install ShellCheck to let MP3 Factory check the configuration file each time the mp3factory.sh script is run, as well as after editing it or loading a different one, provided you do so from the Main Menu. If you don't want to install ShellCheck for some reason then you can test the configuration file on-line at https://www.shellcheck.net/ (you can ignore all the "... appears unused ..." warnings).

usage

Run the script by opening a terminal, navigating to the 'mp3factory' folder, and running ./mp3factroy.sh . The script will run some checks to make sure its dependencies are met, etc., and then create some folders after which it's ready to use. No operations will be performed on your files without your explicit input and all important operations are logged for later reference.

During the course of running tasks you will be asked questions and given choices such as, for example, 'Do you want to do this? [Y/n]'. The default choice is indicated by the upper case letter and can be selected either by pressing the letter, 'y' for 'yes' in this instance, or 'Enter'/'Return', or any other key except 'n'.

If something goes haywire and the script crashes, or you forcefully terminate it by pressing Ctrl+C, and you were working on a file when the problem occurred, the file will be named 'working.*' should you want to recover it.

When processing your files you can process one at a time, pausing before moving on to the next, or you can batch process all of them. Batch processing is activated when you disable pausing when starting a task. If you are new to MP3 Factory it is recommended to avoid batch processing until you become familiar with how the script operates given your configuration. When files are batch processed they will be automatically retained or discarded based on these options, whereas when batch processing is not active you may override the automatic handling of the files. Discarded files are moved to the /junk folder by default and their names are written to the junk.txt file should you want to locate better copies.

You can work on multiple file types all at once, however only the file types which are compatible with the chosen task will be processed. If you're new to MP3F, and perhaps even if you aren't, you may find it easier to work on only one file type at a time.

While the order of the tasks in the MP3 Factory main menu may seem odd, there are logical reasons for it. When performing multiple tasks they should generally be performed in order, though there are exceptions. For example, one probably wouldn't want to rebuild a music file without a proper file name before writing a correct file name based on the metadata that will be forever lost when performing a Rebuild task (assuming the default settings). Similarly, it's probably pointless to invoke most tasks following the first Integrity Check task before an Integrity Check is performed.

Using the Tag To File Name task as a final example, it is first in line for good reason. Let us assume the file name '01 - Only Time.mp3'; if you first create a spectrograph image of the audio, '01 - Only Time.mp3.png', and later write a proper file name from the metadata, 'Enya - Only Time.mp3', the base file name in the /spectro directory wouldn't match that of the music file in the /output directory, thus making it more difficult than necessary to later associate the two files. Conversely, tasks from number 12 on can be performed at any time, in any order, and either of the Integrity Check tasks can be performed at any time as well, though you surely wouldn't want to run the second Integrity Check task in batch mode until the necessary prior steps were performed.

It is primarily at the two Integrity Check stages where files are retained or discarded based upon many tests and since this testing relies heavily upon several tools that work primarily with MP3 files, you will probably want to transcode all non-MP3s to MP3 before running a final Integrity Check.

The View Spectrograph task, while somewhat time consuming, can be important in determining whether to keep or discard a file. For some hints in this regard, see the 'view spectrograph' section below.

Begin by copying whatever files you want to process to the 'input' folder and then, from the main menu of MP3 Factory, select a task to perform. The alert person you are, you noticed immediately that i used the word "copy" and not "move and potentially f'up all 40,000 of your irreplaceable music files", right? The point being that you should work on copies of your source files rather than the originals (and you should retain the originals when possible because... reasons).

When adding a comment tag to an MP3, the comment that is written depends upon whether batch mode is enabled. If batch mode is not enabled then you will be prompted for the comment text, whereas if batch mode is enabled, a default comment is written without prompting. The latter is potentially useful for writing the MP3 Factory version and current date as a comment should a new version of MP3F be released with additional or changed features and you decide to re-process your collection. You could also match the date in the comment tags to the date of the archived log files in order to see exactly what settings were used to process the files in question.

After processing a file there may be one or more error messages starting with 'ERROR', 'WARNING' or 'NOTICE'. As already mentioned, you always have the option to override any errors and keep the file when not operating in batch mode, however when batch mode is enabled all files with an error level of 'ERROR' or 'WARNING' are automatically discarded. While files with only a 'NOTICE' level error are retained since they are quite fixable.

Any file with any error that is retained when not operating in batch mode will have it's file name prefixed with '[X]' so that you know further action is necessary. The prefix will be removed if the error is fixed and it passes the second Integrity Check.

If your music collection consists of singles you downloaded from all over the damn place (like i do) without paying much attention to quality, then don't have a stroke when a huge chunk of it is flagged as junk when subjecting it to an Integrity Check with the default settings. A great deal of music on the interwebs was encoded by dummies like my somewhat former self, not that i gained many IQ points since then. For example, allot of music is encoded using a constant bit rate (CBR) of 320 kib/s which makes little to no sense since the primary purpose of the MP3 format is to reduce file size. Yes, that's a not-so-gentle poke in the ribs for you "but mah 320 CBR" peeps out there who should instead be using LAME's --preset insane , --preset extreme or --preset standard settings which have been thoroughly tested and which produce transparent results for most people; 'transparent' meaning that they cannot tell the difference between a high quality VBR encoded file and the original. Also, a 320 CBR can actually be of lessor quality than a high quality VBR.

batch processing

If you choose to disable pausing after each song is processed then MP3 Factory will batch process all files in the /input directory that are compatible with the chosen task. In this mode of operation the files will be automatically discarded (moved to the /junk folder by default) or moved to the /output folder depending upon the options you have chosen, as well as other internal parameters. If you're just getting to know MP3F then i suggest/warn/demand you avoid batch processing until you become familiar with it.

view spectrograph

MP3 Factory uses ffmpeg to create PNG images of the frequency spectrum of your music files. If batch mode is not enabled you will be shown the image as each image is generated in order to make an immediate determination as to whether to keep the song. If batch mode is enabled then no preview is shown but the images are still written to the /spectro folder with the same base file name as the song (i.e. 'this - song.mp3' has the corresponding image file name of 'this - song.png').

One of the things to look for in these images is the frequency cutoff point, meaning the highest frequency the audio attains. The highest frequency that can be encoded in an MP3 is limited by the bit rate, however there is more to consider. Another thing to look for is clipping. Pictures are worth stuff they say, so let's give that a shot...

Here's what the frequency spectrum looks like for a random FLAC music file that indicated a bitrate of 872 kbit/s. The keen observer you are, you'll notice that the frequency cutoff is close to 21000 Hz, or 21 kHz, which is slightly higher than the highest frequency the average young human is capable of hearing (20 kHz). The files size here is a whopping 26,924,696 bytes.

Frequency spectrum for FLAC sample 872kbps

Next i transcoded the 872 kbit/s FLAC file to a 320 kbit/s CBR MP3. The file size dropped to 9,882,876 bytes, less than half of what it was, yet we lost very little quality. You'll notice the frequency cutoff point has dropped to around 20000 Hz, or 20 kHz, so we lost a little something, but nothing anybody without very terribly keen hearing in a quiet environment with a decent sound system is likely to notice. We also saved a hell of a lot of disk space which is crucial if you want to cram as many songs as possible onto that microscopic SD card that you struggle to plug in to your fondleslab.

Frequency spectrum for MP3 sample 320kbps

Next i converted the 872 kbit/s FLAC to a 128 kbit/s MP3. This time the quality loss is significant to the point where most people with decent hearing in a quiet listening environment and a decent sound system would probably notice. If you're listening to music while operating a jackhammer however, then a 128 kbit/s might be just dandy. Here the frequency cutoff is around 16000 Hz, or 16 kHz, and the file size is 3,953,289 bytes, again half of what it was.

Frequency spectrum for MP3 sample 128kbps

Now i 'did as stoopid does'; i took the 128 kbit/s MP3 and upsampled it to a 320 kbit/s because "more quality", however the only thing i actually accomplished was to fatten the file size, which has now more than doubled to 9,882,876 bytes, the same size of the 320 kbit/s MP3 that was converted from FLAC earlier. The quality is, at best, no better than the 128 kbit/s file and the frequency cutoff point hasn't changed. The lesson here is that you cannot add quality to an MP3 file; it either exists in the first place or it never will.

Frequency spectrum for MP3 sample 320kbps

So now we know that files with a high bit rate and a low frequency cutoff point are upsampled junk, right? Well, no, not necessarily.

When encoding an MP3, audio data is sacrificed in order to reduce the file size, however the way that a good encoder (LAME) does this is to discard sound that the ear would never hear anyway, thus everything above and below a given frequency is discarded right off the bat and it doesn't end there. The bit rate is a primary factor regarding sound quality and frequency cut-off with the LAME encoder. As the bit rate is decreased, so too is the maximum frequency that the audio is able to attain. We know that the average young human with really good hearing can hear frequencies up to about 20 kHz, but as we age that threshold drops. Below is the approximate maximum frequency that can be attained at a given bit rate for a LAME encoded MP3. Keep this in mind when viewing spectrograph images of your audio files:

64  kbit/s = ~11 kHz frequency cut-off
128 kbit/s = ~16 kHz frequency cut-off
192 kbit/s = ~19 kHz frequency cut-off
320 kbit/s = ~20 kHz frequency cut-off

But again, the frequency cut-off point is not in itself a determining factor of file quality. While the cut-off points above are useful for Rock and Pop type genres with a variety of instruments and vocals, they may be meaningless for songs which are not composed with so much diversity. For example, a 'perfect' quality audio recording of a flute, a vocal, or a piano, may have a frequency cut-off point that is well below those stated for a given bitrate.

The other thing to look for in the spectrographs is signs of clipping. Clipping is the result of the gain (volume) having been raised too high and this can lead to really nasty distortion when listening to the song. This is not uncommon given the loudness war we've been subjected to, as well as poor encoding practices. Here is a frequency spectrum of Acoustic Alchemy - Clear Air For Miles.mp3 (320 kbit/s CBR 44.1 KHz). There are no significant signs of clipping here:

Frequency spectrum: Acoustic Alchemy - Clear Air For Miles MP3

Using MP3gain i then increased the gain by 10 dB and this was the result:

Frequency spectrum: Acoustic Alchemy – Clear Air For Miles MP3 (clipping)

What you'll notice is that a lot of the green and blue stuff now reaches the upper frequency cutoff point and this is indicative of clipping.

So how do you use all this information? Well, this is where it gets tricky because an MP3 that was encoded using a high bit rate, yet doesn't approach 20 kHz, is not necessarily of poor quality. Given the wide variety of sounds in most Rock, Pop and some other genres of music we listen to today however, these songs will often approach or slightly exceed 20 kHz even after being converted to MP3 as long as a high quality VBR or CBR encoding method was used.

Over time you'll develop a sense of what the frequency spectrum should look like given the bit rate and the different kinds of sounds present in the recording. Ultimately what matters however is whether you're happy with what you hear, not the fancy colors on a graph.

See also: Spectral Analysis (archive) and Audio Checker – Check the True Quality of Your Audio Files

integrity check

The Integrity Check operations are fairly comprehensive, therefor i think they deserves special attention so you can understand what's going on under the hood and why all your music gets flagged as junk. Actually i'm just in the mood to elaborate about technical crap in such a way as to appear as though i know what the hell i'm talking about, but we'll ignore that.

When running an Integrity Check many tests are performed depending on the options you've defined. When batch mode is not enabled you always have the option to keep or discard the file, whereas when batch mode is enabled, the fate of the file depends entirely upon those options. Here's an overview of what happens when you process an MP3...

  1. ffprobe spits out a pile of information about the file, though it is not used for any purpose other than informational. This output is disabled by default.
  2. A check is performed to see that the tags (artist, title, etc.) you require are present.
  3. eyeD3 spits out a bunch of information regarding the ID3 metadata and a test is performed to see whether the amount of ID3 padding exceeds a given value.
  4. mpck is run and several tests are performed:
    1. Check that the ID3 tags are of an acceptable version.
    2. Check for the presence of obsolete APEv1/2 tags.
    3. Report the integrity of the file.
  5. mp3vl is run to check for various issues.
  6. mp3info is run and a bunch of tests are conducted to determine the integrity of the file, including:
    1. Whether the sample rate is acceptable.
    2. Whether the average bit rate is acceptable.
    3. Whether there are any damaged audio frames.
    4. Whether the MPEG version is version 1.x.
    5. Whether the MPEG layer is layer 3.
    6. Whether the audio channel layout is acceptable (stereo, joint stereo, mono, etc.).
    7. Whether the file was encoded from the original media.
    8. Whether the encoding method is acceptable (VBR, CBR).
    9. Whether the duration is acceptable (helps to weed out truncated or concatenated files).
    10. Whether the audio frames include a CRC checksum, which is probably a bad idea (see the 'CRC verification' section below).
    11. mp3guessenc is run and more tests are conducted to determine whether the file is good or bad (have you realized yet just how hard i'm trying to flag all your music as garbage?). These tests include:
      1. Whether a LAME tag is present and, if so...
        1. Whether the CRC checksum in the LAME tag matches a hash of all the audio frames. This can ('can' being emphasized) tell you whether someone or something mucked with the actual audio, such as mp3gain, mp3splt, etc.).
        2. Whether 'unwise' encoder settings were used.
        3. Whether the 'nspsytune' option was used with LAME.
        4. Whether the 'nssafejoint' option was used with LAME.
        5. Whether the 'lowpass' filter setting is acceptable.
      2. Whether the file size overhead is of an acceptable value or, in other words, whether there is too much metadata or other garbage.
    12. ffmpeg is run to check for decoding errors.

If the file is not an MP3 then only a subset of the tests can be performed. For instance for a FLAC file the following tests are performed:

  1. ffprobe spits out a pile of information, though this information is not used for any purpose other than informational.
  2. A check is performed to see that the tags (artist, title, etc.) you require are present.
  3. metaflac is run and the information it provides is used to determine whether the file is good or bad. The tests performed are:
    1. Whether the sample rate is acceptable.
    2. Whether the number of bits per sample is acceptable.
    3. Whether the duration is acceptable (helps to weed out truncated or concatenated files).
    4. Whether the number of audio channels is acceptable.
  4. ffmpeg is run to check for decoding errors.

CRC verification

The LAME encoder v3.90 or newer writes CRC checksums to the LAME/Xing header of the MP3 when it's encoded. While not bulletproof, this checksum can be used to verify the integrity of the audio stream as a whole (metadata such as artist, title, etc., do not affect the audio verification). While LAME can also write a CRC checksum to each audio frame which may provide a more accurate diagnosis as to whether the audio has been altered, this is discouraged since the space used for this information is the same as that used for the actual audio, thus leading to a possible reduction in audio quality. MP3 Factory can use mp3guessenc to verify that the checksum in the LAME tag matches a hash of the audio and while enabling this check is a great way to weed out a lot of damaged, truncated, or otherwise compromised files, be aware that MP3s processed by any tool which modifies the audio frames, such as mp3splt and mp3gain, may fail the CRC check. Also MP3s that were encoded using ffmpeg will not include this information and therefore the CRC check will fail.

trim silence

Trimming silence should be avoided whenever possible, however if it is necessary it is best to perform the operation on a FLAC file before it is converted to an MP3, else it will not pass an Integrity Check given the default options, though this can be safely ignored. If it is an MP3 that must be trimmed, it is possible it wasn't properly encoded, or the music player (the decoder) isn't discarding the silence added by the encoder (typically LAME). Furthermore if there is a CRC checksum in the LAME tag it will no longer match that of the audio and thus, again, it will no longer pass an Integrity Check, though this can be safely ignored.

In summary, if it is necessary to trim silence from the file, it is best to locate a better copy whenever possible.

Mp3splt can trim more than silence if the silence detection threshold (the value of -p th in the Mp3splt options) is not properly set, therefore you should never batch trim all your music files! It is always best to preview them first to determine if they actually need trimming. When pausing is enabled you will have the opportunity to preview a few seconds of the beginning and end of each file both before and after trimming and, if you're not satisfied with the result, you can adjust the silence threshold and start over. If you find that not enough silence was removed, lower the db threshold by 1 or 2 (e.g. from -48 to -50) and repeat the process. Conversely, if too much was removed, raise the threshold and try again. Don't be surprised if, on occasion, you run across a file where you simply cannot trim it the way you want regardless of the silence threshold value. Again, it's probably better to locate a better copy than fool with silence trimming.

repair

The repair tasks should not be invoked carelessly. The 'Rebuild file with ffmpeg' task is intended as a last resort to recover undamaged audio from an otherwise irreparable MP3 (damaged audio can not be repaired). This task will discard all metadata by default. If you really want the smallest possible file size and do not care about the metadata, you could invoke this task to achieve such a result, but be aware that the decoder may not display any song information.

miscellaneous notes

MP3 Factory does not use ffmpeg to encode MP3s, rather it uses LAME directly so that a proper LAME tag with accurate information is present.

ffplay, a part of the ffmpeg package, is used in various operations to preview the music files. ffplay does not offer any graphical controls in its interface but you can control it with hotkeys. See the 3.6 While playing section of the ffplay documentation. The basic controls you may find handy are:

  • space bar : play/pause
  • left/right (arrows) : seek backwards/forwards 10 sec.
  • down/up (arrows) : seek backwards/forwards 1 min.
  • escape key : quit

download

DISCLAIMER: MP3 Factory is in the early stage of development and is considered 'alpha' software, meaning there be bugs here. It may not be feature complete and some stuff may simply fail to work. If you choose to use it you should be competent enough to review the code and smart enough to backup your music files before processing them. If anything explodes, a mirror will reveal who's to blame :)

Having said that, i am developing and using it on my daily driver box without any guardrails. Then again, i've been known to do stupid things that result in catastrophe, like that time i formatted the wrong partition without first making a fresh backup (i never even thought to ask the NSA to send me their copy).

Code repository: https://codeberg.org/12bytes.org/mp3factory
ZIP archive: https://codeberg.org/12bytes.org/mp3factory/archive/main.zip
TAR.GZ archive: https://codeberg.org/12bytes.org/mp3factory/archive/main.tar.gz
Change log: https://codeberg.org/12bytes.org/mp3factory/commits/branch/main

suggested software

MP3 Factory does not depend on the following programs, however i readily recommend them.

Kwave Sound Editor (pkg. name: kwave): Kwave for the KDE desktop is a nice and simple sound editor that i used while developing MP3 Factory. If you need more power, try Audacity or ardour.

MP3 Diags (pkg. name: mp3diags): A very comprehensive MP3 diagnostic and repair utility with an odd, not so intuitive interface, and a bit of a steep learning curve. You probably want the "unstable" version if you can get it to run. You could certainly use MP3 Diags in place of MP3 Factory, but you'll need to do some serious reading, configuring and experimentation. I used MP3 Diags while developing MP3F for testing purposes and to verify it was producing clean MP3s (it is, depending the configuration).

puddletag (pkg. name: puddletag): Used to edit MP3 metadata and more, puddletag is a powerful and excellent replacement for the well received Windows program, Mp3tag.

Sonic Visualiser (pkg. name: sonic-visualizer): An excellent tool to analyze audio in different ways.

resources