Do you use "parenting" ?
Not the links are building a site-map!
Every page you want to show up in the site map needs a "parent" -structure upwards till the "root".
This is not necessarily the start page - you can have more "page trees" on a site… perhaps someone inside as category - or other way round only the start-pages of a category connected up to a parent-page - to show in a site map the "structure" of your site and not the single entry pages of a "blog" or "news" category…
Necessary is the "parenting" ( set up at the bottom of the pages in the +"options" —> "parent" for every page.
This gives you automatically the "breadcrumb navigation" too.