tiene or tiene usted as formal question

Perhaps it was the placement of the words. Although it is not technically incorrect to say "tiene usted un bolígrafo", it is more common to put the pronoun before the verb. So maybe it would be marked correct if you said "Usted tiene un bolígrafo." 

You conjugated the verb correctly in the usted form, so that would not be the issue.

Hope this helped!