Is putting a tsp. of brown sugar in my oatmeal going to kill my diet?

So it's a new year and my goal for this year is to get back into running daily which I've already done, and in the process drop the last of that stubborn weight that is hard to lose. I've put myself on a strict diet....counting calories, high protein, and minimal refined sugars. How much harm can that tsp. of brown sugar in the morning cause?