You will notice the difference...If your remotly interested in performance...either done do it...or Cut down on it.
you'd be surprised how much better the car feels by pulling out the rear seat and panels...Feels a fair bit quicker and handling is better.
Adding a person to the boot is gonna slow you up considerably. But its not the end of the world.