According to コトバンク, 旗色が悪い can be used when there is some connotation of a battle, war or fight - which makes sense because it literally translates to something like the color of the national flag is bad.
Just wondering how natural it would sound in more general contexts of failure or despondency. For example:
私の商売は、大損失で旗色が悪くなった。