Not 100% sure if it'll damage the car or not, but I assume there is a reason for it... from working on car's previously, the suspension parts are usually very specific in the way they need to be round. Guess its to do with the stress it gets put under!! For the sake of a 10minute job it shouldn't be skipped.
Besides if your'e adjusting the camber the tracking might need to be re-done anyway.... even then, it shouldn't knock it out by much just by switching round one bolt!