The Kanji themselves don't suggests gender, besides that 美 is a slightly feminine concept, but I don't believe I've ever heard 美人 referring to a man. The words 美男 and 美女 exist, which suggests to me that 美人 should refer to both.
Here is a quoted text from this article on Wikipedia:
The prefix bi (美) more often than not refers to feminine beauty, and bijin, literally "beautiful person", is usually, though not always, used to refer to beautiful women**.
**Source: Buckley (2002). Encyclopedia of Contemporary Japanese Culture. Taylor & Francis. pp. 188, 522, 553. ISBN 0-415-14344-6.
Google Dictionary (Japanese) includes beautiful men in its definition.
If you do a google search of 美人, it comes up with women almost exclusively. Which would support the above statement about its mostly referring to women. So, there is evidently a feminine connotation.
This blog also addresses the question:
There are a few ways to praise someone’s attractiveness in Japanese [...]
美しい・うつくしい・utsukushii means 'beautiful’ but when used as an adjective it isn’t often used for people. Therefore if you want to say a girl is beautiful it would be more natural to use the noun 美人・びじん・bijin, as in あの子、美人だ！・あの こ、びじん だ！・ano ko, bijin da! 'that girl over there is beautiful!’. Used a lot more in Japanese than in English, so it is probably more naturally said in English as 'she is so hot’ or something similar, but 美人 is neutral and not overtly sexual. This is pretty much the best anyone can ever look.
イケメン・ikemen is the male variant of 美人 but it’s more of a slang term. Guys can be 美人、but in spoken language it is more natural to call them イケメン。This is basically the equivalent of a 'hot guy' [...]
To sum up the answer on that site: You can refer to both males and females as 美人 but it's less common with men and you would tend to use a different expression like イケメン.
Dictionary says the word was also used to refer to a man:
美しい容貌の女性。美女。麗人。 〔古くは、男子もさした。「玉のやうなる美人、…もらひまして聟にいたします／浮世草子・胸算用 2」〕
But in modern sense it's exclusively used for women and girls.