“Our object must be to bring our territory into harmony with the numbers of our population.”