It's already good that you observe the effect you were expecting in the averages.
Maybe you could filter a bit your signal, that could help observing more stable differences, similarly to what was done in this study:
https://neuroimage.usc.edu/brainstorm/Tutorials/VisualGroupOrig#Subject_averages:_Filter_and_normalize
Do not use blindly this arbitrary 32Hz, try to use something similar to the literature in this domain.
do some permutation across all the vertexes of the ROI and time points
Permutations between nearby vertices or time points are not an interesting test: we expect these measures to be very similar, and consider them as "multiple comparisons" in the statistics. You'd rather average the vertices within your ROI and the time points within your time windows of interest, and then test across subjects (if you have multiple subjects) or across trials (for a single subject study).
Additional guidelines for the statistical analysis:
https://neuroimage.usc.edu/brainstorm/Tutorials/Workflows
https://neuroimage.usc.edu/brainstorm/Tutorials/VisualGroupOrig