Punctuation character at the end of Hebrew text is "shift" to the right if a SHIFT+ENTER (line break within the same paragraph) was typed somewhere in the paragraph.

A simple example is shown in the attached image.

Also, in the attached Image I am using the "Adobe Hebrew" font with "Hebrew" set as Language (as can be seen in the attached images).