Your idea is something which has been studied pretty intensively for decades; it's that there are huge and distinct regions of antimatter with vanishing baryon number B = 0 and that these have existed since at least the start of early structure formation, thus hoping to avoid (highly redshifted) signals in the cosmic diffuse gamma background.
That the anisotropies in the CMB are small puts sufficient limits on structure formation in the earlier universe that our observations (even at COBE scale, let alone from recent observatories) preclude your idea. AMS 02 results to date essentially preclude the wiggle-room of very large antimatter structures at large comoving distances which have "always" been spacelike separated from very large matter structures. BAO observations also put strong limits on the existence of large antimatter structures; it is very very hard to think of ways that gravitational collapse of antimatter can avoid leaving obvious traces on the Lyman-\alpha forest, without entirely overthrowing the Copernican principle.
As far as we can tell -- and there are thirty years of observations probing exactly this question -- there are zero antimatter AGNs and zero early superluminous antimatter supernovae in our sky; if they exist at all they are outside our Hubble volume, or there is a strongly Earth-focused conspiracy about the directions in which AGN relativistic jets point and the alignments on which they are filamented, which is also in tension with the Copernican principle, and that still does not explain the absorption lines on early SN emissions.
This does not preclude other Hubble volumes filled with antimatter or with annihilation products; most models of inflation lead to many many Hubble volumes. However the observations to date effectively preclude much antimatter, and essentially all large structures of antimatter, in our own Hubble volume.
There are some further arguments and an overview of many years worth of the large-antimatter-filled-regions-in-our-universe arguments in astro-ph/9707087 (Glashow et al., 1998)
It was never a stupid argument, but it received a lot of attention in the past couple of decades, and is now essentially resolved.
(There's an overview and citations of more recent discussion in part II of https://arxiv.org/abs/1204.4186v2 although the paper also discusses other ideas about "why isn't there more antimatter in our Hubble volume" and advances a pretty speculative idea of the authors' own.)
That the anisotropies in the CMB are small puts sufficient limits on structure formation in the earlier universe that our observations (even at COBE scale, let alone from recent observatories) preclude your idea. AMS 02 results to date essentially preclude the wiggle-room of very large antimatter structures at large comoving distances which have "always" been spacelike separated from very large matter structures. BAO observations also put strong limits on the existence of large antimatter structures; it is very very hard to think of ways that gravitational collapse of antimatter can avoid leaving obvious traces on the Lyman-\alpha forest, without entirely overthrowing the Copernican principle.
As far as we can tell -- and there are thirty years of observations probing exactly this question -- there are zero antimatter AGNs and zero early superluminous antimatter supernovae in our sky; if they exist at all they are outside our Hubble volume, or there is a strongly Earth-focused conspiracy about the directions in which AGN relativistic jets point and the alignments on which they are filamented, which is also in tension with the Copernican principle, and that still does not explain the absorption lines on early SN emissions.
This does not preclude other Hubble volumes filled with antimatter or with annihilation products; most models of inflation lead to many many Hubble volumes. However the observations to date effectively preclude much antimatter, and essentially all large structures of antimatter, in our own Hubble volume.
There are some further arguments and an overview of many years worth of the large-antimatter-filled-regions-in-our-universe arguments in astro-ph/9707087 (Glashow et al., 1998)
It was never a stupid argument, but it received a lot of attention in the past couple of decades, and is now essentially resolved.
(There's an overview and citations of more recent discussion in part II of https://arxiv.org/abs/1204.4186v2 although the paper also discusses other ideas about "why isn't there more antimatter in our Hubble volume" and advances a pretty speculative idea of the authors' own.)