(Rewritten, although not that different from original wording, see edit history)
The claim was made by Frazer in The Golden Bough, Chap. 49 section 3 "Attis, Adonis, and the Pig", last §:
The Greeks could not decide whether the Jews worshipped swine or abominated them. On the one hand they might not eat swine; but on the other hand they might not kill them. And if the former rule speaks for the uncleanness, the latter speaks still more strongly for the sanctity of the animal.
[...]
We are confirmed in this opinion by observing that down to the time of Isaiah some of the Jews used to meet secretly in gardens to eat the flesh of swine and mice as a religious rite. Doubtless this was a very ancient ceremony, dating from a time when both the pig and the mouse were venerated as divine, and when their flesh was partaken of sacramentally on rare and solemn occasions as the body and blood of gods. And in general it may perhaps be said that all so-called unclean animals were originally sacred; the reason for not eating them was that they were divine.
Frazer did not back it with any bibliographical reference. I do not have his Folklore in the Old Testament, so I cannot check if it is a theory he developed at length there. I thought first he relied on Isaiah 65:3-4; however the claim, almost as worded in the title of this post, is in Isaiah 66:17:
They that sanctify themselves, and purify themselves in the gardens behind one tree in the midst, eating swine's flesh, and the abomination, and the mouse, shall be consumed together, saith the LORD.
The KJV (quoted above) translates literally the Latin of the Vulgate and the Greek of the LXX: if quoting from Frazer did not prove notability, this settles it I think. I cannot read the Masoretic Text so no way to check if it agrees at the literal level.
The only difference between Frazer's claim (1920 CE) and the one by the LXX ( ≤ mid-2nd C. BCE) is: Frazer regards the eating of pork & mice as clandestine religious rituals. Strictly speaking, Isaiah 66:17 says enough Jews upkeep sacred groves and eat non-kosher food that the Lord will come down tough on the whole nation: not that they eat non-kosher food in sacred groves.
In the 1st wording of this post, I quoted Isaiah 65:3-4, the bold part of which seems vaguely relevant to Frazer:
A people that provoketh me to anger continually to my face; that sacrificeth in gardens, and burneth incense upon altars of brick;
Which remain among the graves, and lodge in the monuments, which eat swine's flesh, and broth of abominable things is in their vessels;
However, this has more to do with upkeeping private altars and with necromantic practices.
To clarify, I am not interested in the validity of what these verses claim: I do not read them as accusing all the Israelites of ritually eating swine meat and unclean broths in sacred gardens. There is no dispute that pork was consumed in the kingdom of Judah, so why would some Jews not taste of it and why, then, would they throw away flour in which they found mouse hairs (if that is what "eating the mouse" was actually about).
I just guess Frazer had them in mind when he wrote this § of The Golden Bough and he expected his readers to have them in mind as well, although he did not say so anywhere in the book IIRC.
As for "the time of Isaiah": Isaiah's dates are usually given as late 8th - early 7th C. BCE; those who dispute he authored chapters ≥ 49 of his Book date them from after the Exile (early to mid-6th C.)
Is there any published work, by Frazer himself or other scholars, to back his claim; or is it unsubstantiated speculation on Isaiah 66:17 and possibly 65:3-4?
Again, that the claim is speculation by Frazer is not really in dispute and it is irrelevant if it is backed by scholarly speculation or scholarly what not: the crucial word is unsubstantiated.