When logging out as a User from the Login As feature, Salesforce signs-out the user completely. It does not return to the Administrator User from which the user originally logged in.
Check that the Force relogin after Login-As-User is enabled.
This setting can be found by navigating to: Setup --> Security --> Session Settings --> Force relogin after Login-As-User. By default, this is enabled in all new Orgs after Summer 2014.