This is going to be a really long post. This is not my work, but it explains it way better than I ever could. If you study the texts from the bible, you will see that in fact Jesus did not have brothers or sisters as many Protestants claim.
A group of persons closely connected with the Saviour appears repeatedly in the New Testament under the designation "his brethren" or "the brethren of the Lord" (Matthew 12:46, 13:55; Mark 3:31-32, 6:3; Luke 8:19-20; John 2:12, 7:3-5; Acts 1:14; 1 Corinthians 9:5). Four such "brethren" are mentioned by name in the parallel texts of Matt 13:55 and Mark 6:3 (where "sisters" are also referred to), namely, James (also mentioned Galatians 1:19), Joseph, or Joses, Simon, and Jude; the incidental manner in which these names are given, shows, however, that the list lays no claim to completeness.
Two questions in connexion with these "brethren" of the Lord have long been, and are still now more than ever, the subject of controversy: (1) The identity of James, Jude, and Simon; (2) the exact nature of the relationship between the Saviour and his "brethren".
(1) The identity of James, Jude and Simon. James is without doubt the Bishop of Jerusalem (Acts 12:17, 15:13, 21:18; Galatians 1:19; 2:9-12) and the author of the first Catholic Epistle. His identity with James the Less (Mark 15:40) and the Apostle James, the son of Alpheus (Matthew 10:3; Mark 3:18), although contested by many Protestant critics, may also be considered as certain. There is no reasonable doubt that in Galatians 1:19: "But other of the apostles [besides Cephas] I saw none, saving James the brother of the Lord", St. Paul represents James as a member of the Apostolic college. The purpose for which the statement is made, makes it clear that the "apostles" is to be taken strictly to designate the Twelve, and its truthfulness demands that the clause "saving James" be understood to mean, that in addition to Cephas, St. Paul saw another Apostle, "James the brother of the Lord" (cf. Acts 9:27). Besides, the prominence and authority of James among the Apostles (Acts 15:13; Galatians 2:9; in the latter text he is even named before Cephas) could have belonged only to one of their number. Now there were only two Apostles named James: James the son of Zebedee, and James the son of Alpheus (Matthew 10:3; Mark 3:18; Luke 6:16; Acts 1:13). The former is out of the question, since he was dead at the time of the events to which Acts 15:6 ssq., and Galatians 2:9-12 refer (cf. Acts 12:2). James "the brother of the Lord" is therefore one with James the son of Alpheus, and consequently with James the Less, the identity of these two being generally conceded. Again, on comparing John 19:25 with Matt 27:56, and Mark 15:40 (cf. Mark 15:47; 16:1), we find that Mary of Cleophas, or more correctly Clopas (Klopas), the sister of Mary the Mother of Christ, is the same as Mary the mother of James the Less and of Joseph, or Joses. As married women are not distinguished by the addition of their father's name, Mary of Clopas must be the wife of Clopas, and not his daughter, as has been maintained. Moreover, the names of her sons and the order in which they are given, no doubt the order of seniority, warrant us in identifying these sons with James and Joseph, or Joses, the "brethren" of the Lord. The existence among the early followers of Christ of two sets of brothers having the same names in the order of age, is not likely, and cannot be assumed without proof. Once this identity is conceded, the conclusion cannot well be avoided that Clopas and Alpheus are one person, even if the two names are quite distinct. It is, however, highly probable, and commonly admitted, that Clopas and Alpheus are merely different transcriptions of the same Aramaic word Halphai. James and Joseph the "brethren" of the Lord are thus the sons of Alpheus.
Of Joseph nothing further is known. Jude is the writer of the last of the Catholic Epistles (Jude 1). He is with good reason identified by Catholic commentators with the "Judas Jacobi" ("Jude the brother of James" in the Douay Version) of Luke 6:16 and Acts 1:13, otherwise known as Thaddeus (Matthew 10:3; Mark 3:18). It is quite in accordance with Greek custom for a man to be distinguished by the addition of his brother's name instead of his father's, when the brother was better known. That such was the case with Jude is inferred from the title "the brother of James", by which he designates himself in his Epistle. About Simon nothing certain can be stated. He is identified by most commentators with the Symeon, or Simon, who, according to Hegesippus, was a son of Clopas, and succeeded James as Bishop of Jerusalem. Some identify him with the Apostle Simon the Cananean (Matthew 10:4; Mark 3:18) or the Zealot (Luke 6:15; Acts 1:13). The grouping together of James, Jude or Thaddeus, and Simon, after the other Apostles, Judas Iscariot excepted, in the lists of the Apostles, (Matthew 10:4-5; Mark 3:18; Luke 6:16; Acts 1:13) lends some probability to this view, as it seems to indicate some sort of connexion between the three. Be this as it may, it is certain that at least two of the "brethren" of Christ were among the Apostles. This is clearly implied in 1 Cor 9:5: "Have we not the power to carry about a woman, a sister, as well as the rest of the apostles, and the brethren of the Lord, and Cephas?" The mention of Cephas at the end indicates that St. Paul, after speaking of the Apostles in general, calls special attention to the more prominent ones, the "brethren" of the Lord and Cephas. The objection that no "brethren" of the Lord could have been members of the Apostolic college, because six months before Christ's death they did not believe in Him (John 7:3-5), rests on a misunderstanding of the text. His "brethren" believed in his miraculous power, and urged him to manifest it to the world. Their unbelief was therefore relative. It was not a want of belief in His Messiahship, but a false conception of it. They had not yet rid themselves of the Jewish idea of a Messiah who would be a temporal ruler. We meet with this idea among the Apostles as late as the day of the Ascension (Acts 1:6). In any case the expression "his brethren" does not necessarily include each and every "brother", whenever it occurs. This last remark also sufficiently answers the difficulty in Acts 1:13-14, where, it is said, a clear distinction is made between the Apostles and the "brethren" of the Lord.
(2) The exact nature of the relationship between the Saviour and his "brethren". The texts cited at the beginning of this article show beyond a doubt that there existed a real and near kinship between Jesus and His "brethren". But as "brethren" (or "brother") is applied to step-brothers as well as to brothers by blood, and in Scriptural, and Semitic use generally, is often loosely extended to all near, or even distant, relatives (Genesis 13:8, 14:14-16; Leviticus 10:4; 1 Chronicles 15:5-10, 23:21-22), the word furnishes no certain indication of the exact nature of the relationship. Some ancient heretics, like Helvidius and the Antidicomarianites, maintained that the "brethren" of Jesus were His uterine brothers the sons of Joseph and Mary. This opinion has been revived in modern times, and is now adopted by most of the Protestant exegetes. On the orthodox side two views have long been current. The majority of the Greek Fathers and Greek writers, influenced, it seems, by the legendary tales of apocryphal gospels, considered the "brethren" of the Lord as sons of St. Joseph by a first marriage. The Latins, on the contrary, with few exceptions (St. Ambrose, St. Hilary, and St. Gregory of Tours among the Fathers), hold that they were the Lord's cousins. That they were not the sons of Joseph and Mary is proved by the following reasons, leaving out of consideration the great antiquity of the belief in the perpetual virginity of Mary. It is highly significant that throughout the New Testament Mary appears as the Mother of Jesus and of Jesus alone. This is the more remarkable as she is repeatedly mentioned in connexion with her supposed sons, and, in some cases at least, it would have been quite natural to call them her sons (cf. Matthew 12:46; Mark 3:31; Luke 8:19; Acts 1:14). Again, Mary's annual pilgrimage to Jerusalem (Luke 2:41) is quite incredible, except on the supposition that she bore no other children besides Jesus. Is it likely that she could have made the journey regularly, at a time when the burden of child-bearing and the care of an increasing number of small children (she would be the mother of at least four other sons and of several daughters, cf Matthew 13:56) would be pressing heavily upon her? A further proof is the fact that at His death Jesus recommended His mother to St. John. Is not His solicitude for her in His dying hour a sign that she would be left with no one whose duty it would be to care for her? And why recommend her to an outsider if she had other sons? Since there was no estrangement between Him and His "brethren", or between them and Mary, no plausible argument is confirmed by the words with which he recommends her: ide ho uios sou, with the article before uios (son); had there been others sons, ide uios sou, without the article, would have been the proper expression.
The decisive proof, however, is that the father and mother of at least two of these "brethren" are known to us. James and Joseph, or Joses, are, as we have seen, the sons of Alpheus, or Clopas, and of Mary, the sister of Mary the Mother of Jesus, and all agree that if these are not brothers of the Saviour, the others are not. This last argument disposes also of the theory that the "brethren" of the Lord were the sons of St. Joseph by a former marriage. They are then neither the brothers nor the step-brothers of the Lord. James, Joseph, and Jude are undoubtedly His cousins. If Simon is the same as the Symeon of Hegesippus, he also is a cousin, since this writer expressly states that he was the son of Clopas the uncle of the Lord, and the latter's cousin. But whether they were cousins on their father's or mother's side, whether cousins by blood or merely by marriage, cannot be determined with certainty. Mary of Clopas is indeed called the "sister" of the Blessed Virgin (John 19:25), but it is uncertain whether "sister" here means a true sister or a sister-in-law. Hegesippus calls Clopas the brother of St. Joseph. This would favour the view that Mary of Clopas was only the sister-in-law of the Blessed Virgin, unless it be true, as stated in the MSS. of the Peshitta version, that Joseph and Clopas married sisters. The relationship of the other "brethren" may have been more distant than that of the above named four.
The chief objection against the Catholic position is taken from Matt 1:25: "He [Joseph] knew her not till she brought forth her firstborn son"; and from Luke 2:7: "And she brought forth her firstborn son". Hence, it is argued, Mary must have born other children. "Firstborn" (prototokos), however, does not necessarily connote that other children were born afterwards. This is evident from Luke 2:23, and Ex 13:2-12 (cf. Greek text) to which Luke refers. "Opening the womb" is there given as the equivalent of "firstborn" (prototokos). An only child was thus no less "firstborn" than the first of many. Neither do the words "he knew her not till she brought forth" imply, as St. Jerome proves conclusively against Helvidius from parallel examples, that he knew her afterwards. The meaning of both expressions becomes clear, if they are considered in connexion with the virginal birth related by the two Evangelists.