A book about architecture I'm trying to read (it's a bit of a slog what with the technical jargon and such) contained this description of a temple building: 二重、初重５X4間、裳階 （もこし） つき、上重４X3間. So it's two stories, the first being five rooms by four, with lean-to rooms attached, etc. I have a few questions about this. First, how is the "X" in 5X4 read? Additionally, I'm assuming that 初重 and 上重 are read しょじゅう and じょうじゅう, respectively. Is this correct?
The English "by" is read 「かける multiply」 in Japanese. A 3x4 matrix, for example, is 「3かける4行列」 in Japanese. I assume your readings are right in the second part of your question, but can't seem to find them anywhere.
I think you are making a big miskate. As for
５X4間, you interpret it as 'five rooms by four', but it is not clear what that means, and probably that is wrong.
間 is a traditional unit for length. It is approx the long length of a tatami (about 1.8m).
５X4間 means a ５間 by 4間 rectangle.