Doesn't do as much as you think. Instead it might backfire(assuming the state doesn't cut budgets/other revenue sources).
As reported: https://oilprice.com/Energy/Gas-Prices/Did-New-Yorks-Tax-Holiday-Actually-Help-Reduce-Gasoline-Price-Pain.html
An analysis by the Institute on Taxation and Economic Policy (ITEP) for New York Focus showed that less than 50% of the tax savings made its way to consumers. The article estimated that 30% was being retained by the oil industry, which is consistent with the argument I made earlier. However, I will admit I thought it would be more than 30%.
Important things to note. Supply side fixes are the best. But they're also the hardest.