how to fix InteractiveViewer and scrollviews competing for gestures

I have a vertical scrollView then a horizontal pageView containing images wrapped inside InteractiveViewer.

but it's quite a mess when I try zooming-in with the InteractiveViewer. all those scrollviews are competing for the gestures and I was wonding an easier way to get that fixed.

I have something like

CustomScrollView(
  slivers: [
   const SliverToBoxAdapter(
         child: PageView(
         ...
          InteractiveViewer()
         ...
      )
    ),
 ]
)