Answer:

This Shabbos is also Rosh Chodesh, requiring the recital of a special text for the middle bracha of Musaf. This special Musaf includes elements of the usual Shabbos Musaf, the usual Rosh Chodesh Musaf, and a special introductory passage. This passage, beginning with the words Atah Yatzarta, actually bears closer resemblance to the introductory part of the Yom Tov Musaf than it does to Musaf of either Shabbos or Rosh Chodesh. The rest of the middle bracha of Musaf combines elements of both Shabbos Musaf and Rosh Chodesh Musaf.

I once edited an article in which the author quoted several anthologies, each of which ruled that someone who realizes he is saying Tikanta Shabbos on Shabbos Rosh Chodesh should immediately stop where he is, and go to the beginning of Atah Yatzarta, and recite the entire bracha. However, I believe that this ruling is in error. I will explain shortly why I believe that this answer is erroneous.  But first…

I attempted to trace the sources quoted in the article to see if perhaps I was missing some logic or information that I would clarify in the course of my research.

What I did discover was that each source was simply quoting a previous one, and that they all traced to one obscure 19^th century work, which did not explain at all the reason for the ruling. Classic group-think.

I will now explain why I believe this ruling is in error, and what one should do. My major concern is that the approach that these works advocate results in repeating many parts of the shemoneh esrei, and that this repetition constitutes a forbidden interruption in the tefillah. Furthermore, to the best of my knowledge, there is no essential requirement to recite this middle bracha of the shemoneh esrei precisely in order. Obviously, one should maintain the order as is, but there is ample evidence from major halachic authorities that, in general, mistakenly rearranging the order of a bracha is not calamitous (see, for example, Rosh, Taanis 1:1; Shu”t Igros Moshe, Orach Chayim 4:18 and 4:70:14). Thus, when left with the choice of rearranging the order of a bracha to avoid repetition, or repeating parts of the bracha and ignoring what was already said, one should follow the first approach.

Subsequently, I realized that the position I have followed, is indeed that of Rav Moshe Feinstein. However, it appears that, in general, there are other halachic authorities who feel that the text of a brocha should indeed be kept intact even when repetition will result (see, for example, Mateh Efrayim 582:10; Mishnah Berurah 582:16; Biur Halacha 127:2 s.v. Aval).

Notwithstanding the disputing opinion, I still think that the approach I am suggesting is correct, but I recognize that others may disagree with me. Therefore, I am going to present my approach, as confusing as it may appear.

Based on my opinion, it appears that someone who discovers that he/she began reciting Tikanta Shabbos rather than Atah Yatzarta should mention only those parts of the bracha that he/she has as yet not recited, but not repeat any theme or part of the bracha that one has already said. Although fulfilling this may be confusing to someone unfamiliar with the bracha, this should provide us with a valid reason to pay more attention to the details of this bracha and understand its different parts.

In order to explain how one does this correctly, brachos of Atah Yatzarta and Tikanta Shabbos into their constituent parts, so that we can identify which parts we should not repeat. We can divide these brachos into the following seven sections (the sections for a regular Shabbos have been numbered in a way that parallels the list for Shabbos Rosh Chodesh:

Shabbos Rosh Chodesh

Regular Shabbos

1. The introduction – from the words Atah Yatzarta until and including the words shenishtalcha (some recite the text hashelucha) bemikdashecha.

1. The introduction – from the words Tikanta Shabbos until and including the word kara’ui.

2. The prayer for our return – beginning with the words Yehi Ratzon – until (and including) the word kehilchasam.

2. The prayer for our return – beginning with the words Yehi Ratzon – until (and including) the word kehilchasam.

3. The sentence that introduces the mention of the pesukim of the Musaf – Ve’es Musafei Yom HaShabbos Hazeh veyom Rosh Hachodesh… until (and including) the word ka’amur.

3. The sentence that introduces the mention of the pesukim of the Musaf – Ve’es Musaf Yom HaShabbos Hazeh… until (and including) the word ka’amur.

4. Mention of the pesukim of the korban Musaf of Shabbos.

4. Mention of the pesukim of the korban Musaf of Shabbos.

5. Mention of the pasuk of the korban Musaf of Rosh Chodesh and the passage Uminchasam… until (and including) the word kehilchasam.

6. The paragraph Yismechu Vemalchusecha that concludes with the words zeicher lemaasei vereishis.

6. The paragraph Yismechu Vemalchusecha that concludes with the words zeicher lemaasei vereishis.

7. The closing of the bracha -- Elokeinu Veilokei Avoseinu.

7. The closing of the bracha -- Elokeinu Veilokei Avoseinu.

We should note that the closings of these middle brachos of Musaf shemoneh esrei are very different. On Shabbos Rosh Chodesh we recite a version that is almost identical to what we recite on a weekday Rosh Chodesh, but we insert three passages to include Shabbos.

Parts 2, 4 and 6 of the two brachos are identical, whether it is Shabbos or Shabbos Rosh Chodesh. Therefore, one should not repeat these sections if one has said them already.

Part 1 on Shabbos Rosh Chodesh, Atah Yatzarta, is very different from what we usually recite on a regular Shabbos. Therefore, someone still in the middle of this bracha should recite this passage again.

If someone missed part 5, mention of the pesukim of Rosh Chodesh, and is still in the middle of this bracha, he/she should recite it and introduce it with the section 3 above, which introduces the korbanos of the Musaf. However, if he/she already recited the pesukim of Shabbos korban Musaf (#4) above, he should omit the reference to Shabbos in this piece and only mention Rosh Chodesh. In the latter case, one should change the plural Musafei to a singular Musaf since he/she now is only mentioning the Rosh Chodesh Musaf.

Having explained the rules governing these halachos, I will now present the conclusions in a hopefully clearer way, depending on when you discover your mistake:

A. If you were still reciting the beginning of Tikanta Shabbos, and had not yet reached Yehi Ratzon:

Return to Atah Yatzarta and recite it in order without any changes.

B. If you had already begun the Yehi Ratzon, but are before Ve’es Musaf Yom HaShabbos Hazeh:

Complete the Yehi Ratzon until Ve’es Musaf; then recite Atah Yatzarta until the words Yehi Ratzon, then resume from the words Ve’es Musafei Yom HaShabbos Hazeh Veyom Rosh Hachodesh from the Shabbos Rosh Chodesh Musaf and continue through the rest of the tefillah.

C. If you had just begun Ve’es Musaf Yom HaShabbos Hazeh:

Add the words Ve’es Musaf Yom Rosh Hachodesh Hazeh, and then continue in the Shabbos Rosh Chodesh Musaf until Yismechu Vemalchusecha. Immediately prior to saying Yismechu Vemalchusecha insert the words from Atah Yatzarta until the words shenishtalcha bemikdashecha. Then return to Yismechu Vemalchusecha and recite the rest of the tefillah in order.

D. If you are already in the middle of Ve’es Musaf Yom HaShabbos Hazeh:

Recite Uveyom Hashabbas… until Veniskah. Then insert the words from Atah Yatzarta until the words shenishtalcha bemikdashecha. Then return to the words Ve’es Musaf but say the following: Ve’es Musaf Yom Rosh Hachodesh Hazeh until the word ka’amur. Then say Uverashei Chadsheichem in the Shabbos Rosh Chodesh section and continue in order.

E. If you are in the middle of Yismechu Vemalchusecha, complete it until Zeicher lemaasei vereishis, and then insert the words from Atah Yatzarta until the words shenishtalcha bemikdashecha. Then return to the words Ve’es Musaf but say the following: Ve’es Musaf Yom Rosh Hachodesh Hazeh until the word ka’amur. Then say Uverashei Chadsheichem in the Shabbos Rosh Chodesh section. Then go to Elokeinu Veilokei Avoseinu (after Yismechu Vemalchusecha) and finish the end of the bracha and the davening.

F. If you are already in the middle of the closing part of the bracha (Elokeinu Veilokei Avoseinu) complete the clause that you are saying, and then insert the words from Atah Yatzarta until the words shenishtalcha bemikdashecha. Then return to the words Ve’es Musaf but say Ve’es Musaf Yom Rosh Hachodesh Hazeh until the word ka’amur. Then say Uverashei Chadsheichem in the Shabbos Rosh Chodesh section. Then return to chadeish aleinu beyom hashabbos hazeh es hachodesh hazeh and finish the end of the bracha in the Shabbos Rosh Chodesh section.

If you completed the entire bracha of Tikanta Shabbos, but mentioned in the middle of the bracha some reference to the korban Musaf of Rosh Chodesh, you have fulfilled the requirements of this prayer and you should continue Retzei (see Mishnah Berurah 423:6). If you completed the bracha of Tikanta Shabbos but did not yet begin Retzeih, you should say “vena’aseh lefanecha korban Rosh Chodesh hazeh” – “and we shall do before You this Rosh Chodesh offering” and then continue with Retzei (ibid.).

Conclusion

Although all this may sound very confusing, if we spend a few seconds familiarizing ourselves with the divisions of this bracha that I have made, we will easily realize why the halachos are as I have outlined, and will be ready to make the necessary adjustments should we find that we have erred. This readiness has of course a tremendous value on its own: It familiarizes us with the shemoneh esrei, something we always should do, but, unfortunately, often do not pay sufficient attention.

Understanding how much concern Chazal placed in the relatively minor aspects of davening should make us even more aware of the fact that davening is our attempt at building a relationship with Hashem. As the Kuzari notes, every day should have three high points -- the three times that we daven. Certainly, one should do whatever one can to make sure to pay attention to the meaning of the words of one's Tefillah. We should gain our strength and inspiration for the rest of the day from these three prayers. Let us hope that Hashem will accept our tefillos together with those of Klal Yisrael!