I noticed in words like アメリカ and イギリス are written in katakana, but why is 日本 (にほん) in hiragana if Japan is a country too? I noticed katana is also used in modern stuff (like smartphone, スマホ). Was katana made for new words instead of making a whole new complicated kanji?
There is no such rule that says you must write country names in katakana. The correct rule is to write loanwords in katakana. Read the Usage section in this Wikipedia article. Just as English writers sometimes use italics to indicate loanwords (e.g., "wasabi", "ad hoc"), Japanese writers have a rule of using katakana for words derived from languages such as English.
スマホ is written in katakana because it's short for スマートフォン, which is a loanword. Western country names historically don't have corresponding words in Japanese, so katakana is used to represent their sound. On the other hand, some countries that are in the "kanji culture zone" are referred to in kanji rather than katakana. Excluding Japan (日本) itself, the countries that fall into this category are China (中国), South Korea (韓国), and North Korea (北朝鮮, DPRK).
When katakana was first created in the 9th century by Buddhist monks, the purpose was not to write English, but to efficiently notate the reading of sutras that had come from distant Asian countries. Today, we still use katakana primarily for "sound only notation".