Luther was coarse in some of his modes of expression, especially when irritated at aspersions cast on any of his opinions. The fashion indeed seemed to have prevailed at that time to have recourse to low and vituperative terms to enforce conviction either on the party addressed or the world at large. Calvin thought it not beneath him to address Luther in such terms as these: "Your whole school is nothing but a stinking stye of pigs; dog, do you understand me? Do you understand me, madman? do you understand me, you great beast!" [source]
Initial Source: "The Library" by George Crabbe
If you Google search this quote, you'll get hits from a number of 19th Century books, and even a few from the 18th Century. I found what I think to be the source. This book mentions that the quote comes from "the poet Crabbe." This is probably a reference to George Crabbe (1754-1832). In The Poetical Works of the Rev. George Crabbe comes the following 1781 poem, The Library. In the original version is the following:
Calvin grows gentle in this silent coast,
Nor finds a single heretic to roast;
Here, their fierce rage subdued, and lost their pride,
The Pope and Luther slumber side by side.
Against her foes Religion well defends Her sacred truths, but often fears her friends; If learn'd, their pride, if weak, their zeal she dreads, And their hearts' weakness, who have soundest heads: But most she fears the controversial pen, The holy strife of disputatious men ; (1) Who the blest Gospel's peaceful page explore, Only to fight against its precepts more.
(1) ["Your whole school is nothing but a stinking sty of pigs. Dog! do you understand me? Do you understand me, madman? Do you understand me, you great beast ?" — Calvin to Luther.]
Joachim Westphal as Beast and Madman, Not Luther
I've read quite a number of things on Calvin's view of Luther, and I don't ever recall such a statement. Perhaps the best overview I've ever read was B.A. Gerrish, "John Calvin on Luther" found in Interpreters of Luther, Essays in Honor of Wilhelm Pauck (Philadelphia: Fortress Press, 1968) pp. 67- 96. This article is well worth tracking down. Gerrish presents an excellent overview of Calvin's statements about Luther throughout his career. Overall, Calvin's statements in regard to Luther are positive, this despite the controversies between the Lutherans and the Swiss. If this statement was really made by Calvin about Luther, it's indeed quite odd that Gerrish didn't include it.
I searched around a bit more, and it appears the quote in question is not directed to Luther at all, but rather Joachim Westphal. Here is one quote from a number of hits to old books that mention Westphal:
"In very truth," said Calvin," Luther is extremely corrupt., ..(cited by C. Schlusomberg,) would to God that he had been attentive to discover his vices." (Theol. Calv. L. 11. fol. 126.) Calvin elsewhere speaks very contemptuously of the Lutheran Church; (in his reply to Westphal) he says, " Thy school is nothing but a stinking pig-stye; dost thou hear me, thou dog? dost thou hear me, thou mad-man? dost thou hear me, thou huge beast?"[source]Conclusion
In Calvin's Last Admonition to Westphal, Calvin says,
"But who is not horrified at the monstrous blindness of Westphal, who seeks a color for his doctrine from suffrages which might rather cover the sun with darkness? Since he has chosen this vile pig-stye for his school, let him regale himself on the husks which are fit for him: only let the reader remember the proof he gives of his shameful poverty when he is forced to bring his judges from the lowest dregs of the Papacy."Here in this book Calvin is said to have admitted calling Westphal a "beast," and then later retracting the comment. Perhaps somewhere, the entire comment on Westphal exists, but I lack the time to search it out. The interesting thing about Calvin's debates with Westphal is that one of the issues was actually the correct interpretation of Luther, but that's a subject for another time.