I changed my answer after considering Tsuyoshi Ito's answer. As he says, having an affirmative should be preferrable as given in his answer (But that was a different point from your question).
It is grammatical, but may not be completely natural. It is ungrammatical to end it without the latter part. A literal translation is
As for Jim, not to mention whether he is not able to converse in Japanese, he cannot even make simple greetings.
which reflects the unnaturalness in the original Japanese. See Tsuyoshi Ito's answer for the correct sentence, which I translate as
As for Jim, not to mention whether he is able to converse in Japanese, he cannot even make simple greetings.
As the example
shows, the construction does not depend on the existence of a negation in the latter part, so the feeling of denial against the first part seems to come from
どころか. However, also in the makeup of
どころか, there is no morpheme you can attribute the meaning of negation. I take it that the first part of this construction is not negated, but is a neutral proposition, and is excluded from consideration. The English translation will include
whether. Since negation under
whether is redundant, that causes the unnaturalness Tsuyoshi Ito points out.