Like a traditional maths journal, an arXiv overlay accepts submissions of papers from authors, coordinates their peer review by other mathematicians, and ultimately decides which to accept, essentially providing a stamp of approval that the stuff they accept is probably not nonsense. But where a traditional journal then publishes those papers on their website (and some, so I’ve heard, even print them all out every so often, bind them together and mail them to a bunch of Univerity libraries), an arXiv overlay journal lets the arXiv handle that.
The arXiv is a website that hosts preprints of research papers (ie, versions the authors want to make available before they’re formally published). The idea is that since many mathematicians are making their papers freely available on the arXiv anyway before publication, there’s really no need to separately publish them elsewhere once a journal has given them the thumbs-up. So instead an arXiv overlay journal simply provides links to the final revised versions of the papers it’s ‘published’, uploaded at the arXiv (all on a very nice journal website of course).
So why is this needed? Money reasons, of course! Traditional journals charge quite a lot, either to subscribers or authors, for the provision of an arguably redundant service. (This is a matter Gowers has written about before.) They typically don’t pay editors, reviewers or authors, and all that fiddly typesetting is now mostly taken care of by the author’s fluent grasp of LaTeX, so their fees are entirely to cover overhead costs that an overlay journal only incurs a fraction of (and generate profit of course).
Gowers says that the software powering their journal charges $10 per submission, which will be initially covered by a grant from Cambridge University, making the journal entirely free for authors and readers initially with at most a modest fee thereafter.