My friend heard this in an anime. I can understand 複雑な波長パターン or something along these lines, but just 複雑な波長? Like, you wouldn't say "complex mass" or "complex speed", right? (unless maybe complex numbers are involved but that's another thing) Does "complex wavelength" make more sense in Japanese? I get that they wanted to say that the wavelength is difficult to analyze, but is it a correct way to say that?
In this sentence, 複雑な modifies 波長パターン, not just 波長. And you can certainly say "complex pattern", right? Perhaps they're actually analyzing hundreds of wavelength numbers, as you suggested.
As a scientific term, I agree with you; 複雑な波形 (波形 = waveform) is used a lot, for example, to describe the shape of electrocardiograms, but 複雑な波長 makes little sense to me.
But as @YoichiOishi said, 波長 is frequently used metaphorically to express how two people think, feel or act in similar ways ("synchronously"). Ordinary people are probably more familiar with 波長 than 波形.
(If the anime your friend is watching is エヴァンゲリオン, I remember "synchronization" is a keyword of that anime, so somehow 波長 may feel better. That's just a guess, though.)
Though it's premature to say anything without knowing the full context of the given phrase, I think I can surmise the nuance of 複雑な波長 in terms of emotional expression.
We often say "彼（or 彼ら）と僕（or 僕ら）は波長が違う" to mean I (we) disagree with him (them) in the way of thinking, feeling, and opinion. Thus I take 複雑な波長 as a complex pattern of thinking, feeling, character, behavior, or opinion.
I don't know whether 複雑な波長 is appropriate or not in physical expression.