Why was the concept of manifest destiny controversial when it was first outlined?

Manifest Destiny is the belief that Americans had the right to expand westward across the North American continent from the Atlantic Ocean to the Pacific Ocean. The concept of this was controversial when first outlined because some citizens didn't like the idea of "invading" the whole continent. Most people didn't like the idea of change and wanted to remain neutral.

it increase by 12% a year from 1812-1820