By using the selection tools, I can add or subtract areas that I want to include.
This is done by pressing the ALT or Shift keys (a + or - will appear next to your selection tool.
The magnetic lassoo tool is a great one to use as it tries to find the edgest for you. To go back one vector point just press the delete key.