When a Territory filter is selected in My Accounts, Accounts with no Territory or with different Territory are appearing in the view.
This is working as designed. When there is an active sharing rule for the user's Territory that provides access to the Account, the Account appears even if a different territory filter is selected.
Check the Sharing Detail of the Account using the following link:
[Salesforce instance URL]/p/share/AccSharingDetail?parentId=[Account SFDC Id]
The Sharing Settings for Account can be modified in Setup --> Security Controls --> Sharing Settings.
CRM Help Documentation: My Accounts