If these were wasps, you would have had holes in your hand - wasp stings are pretty obviously different from bee stings, in that bees leave the stinger in your skin (they go off and die, alone and buttless), whereas wasps merely puncture you mercilessly until they get tired of it or you run away.
You should definitely call an exterminator. Most wasp nests are unsafe to remove by hand, especially if they're already defensive. Yellowjackets release a hormone when they're threatened that alerts other yellowjackets to danger, as well as making them all more aggressive. This can stay in the air for hours or days, so it'll probably be unsafe to try to move the boards again tomorrow. Hence, calling the exterminator. It's his or her job to put on the moon suit and laugh mercilessly at the helpless bugs as he sprays them with RAID. Not yours.
Finally, in addition to the benadryl (or other anti-inflammatory like hydrocortizone cream), you might consider making a baking soda paste. Mix baking soda in water until it's got a consistency like kindergarten paste, and rub it on the stung area. Part of the joy of wasp stings is that they're acidic, while baking soda...well, it ain't. It will quench the acid and, if made with cool water, provide a soothing poultice. For additional theatrics, place some maple tree leaves (not oak, because they contain tannic acid) over the baking soda paste, grab a plastic sword, and say "I'm George, and I'll slay that dragon if it's the last thing I do!!"

|