No, most Christians, if they actually read, study and know the Bible would not agree that their relationship was never marked once by sexual intimacy.  Nor does it say that Joseph was a widower anywhere in scripture.  That must be an opinion of one of the Catholic "church fathers".   There is absolutely nothing wrong with married folk having sex.  It says in Matthew 1:24-25 That Joseph married Mary, but had no union with her until after she gave birth to her son.  Which indicates to me that they did have union, which is normal in marriage, after she had Jesus.  There are also references in the gospels to Jesus having brothers and sisters.  The book of James was probably written by Jesus's brother.  That is a commonly held opinion of scholars.  The disciple James had already died before this was written, and, according to the NIV study notes, "The other two men named James had neither the stature nor the influence that the writer of this letter had."