Did Maimonides actually ascend the Temple Mount?

I've read the claim in several different places and heard it on various occasions that the Rambam had made a pilgrimage to the Temple Mount. Usually this claim is presented uncritically and unsourced, so I figured I'd try and uncover where this idea came from.

Apparently there is a book that has come to be entitled פירוש הרמב"ם על מסכת ראש השנה that was supposedly copied directly from the handwritten manuscripts of the Rambam by one שמואל בר אברהם שקייל (who I can't seem to find much biographical info about). In several places he the copyist refers to the author as רבינו משה but nowhere, as far as I am aware does he explicitly say that this Moshe is to be identified with the Rambam. At the end of the work there is a letter attributed to this רבינו משה by the copyist which appears to be the source of the legend:

 בליל אחד בשבת בארבעה ימים לחודש אייר נכנסתי לים, וביום שבת, עשירי לאייר שנת תתקכ"ה עמד עלינו נחשול שבים לטבענו והיה זעף גדול בים, ונדרתי עלי: שני ימים אלו אצום בהם ואתענה בהם.. ואצווה על בני לעשות כן עד סוף כל הדורות, ויתנו צדקה כפי כוחם... בליל אחד בשבת, ג' ימים לירח סיוון, יצאתי מן הים בשלום ובאתי לעכו ונצלתי מן השמד והגענו לארץ ישראל. יום זה נדרתי שיהיה יום ששון ושמחה... וביום שלישי בשבת, ד' חשוון, יצאנו מעכו לעלות לירושלים תחת סכנה, ונכנסתי לבית הגדול והקדוש והתפללתי בו. ובאחד בשבת, ט' בחודש חשוון יצאתי מירושלים לחברון, לנשק קברי אבותי במערה... ושני ימים אלו... נדרתי שיהיו יום טוב ותפילה ושמחה לה' לאכילה ולשתייה... וכשם שזכיתי וראיתי בחורבנה כך אזכה אני וכל ישראל לראות בנחמתה, אמן. 

The letter is also replicated in the Sefer Hareidim, written by Elazar Azkiri, a Kabbalist of the Tzefath school (probably most well know for the attribution of the piyyut Yedid Nefesh to him) and introduces its provenance as שהעתיקה הנעלה רבי שמואל בר אברהם שקייל ז״ל בעכו מכתיבת יד הרב רבינו משה .מאור הגולה And from there it would seem to have attained authoritative gospel like status as being intimately tied to the Rambam. The problem is that the attribution of this פירוש על מסכת ראש השנה (and thus the letter) to the Rambam is quite probably mistaken. Menahem Mendel Kasher attempted to demonstrate that it is an erroneous attribution and raises a host of objections. He points towards major contradictions between the peirush and the Mishneh Torah the character of which he argues the Rambam would have rectified had he had a change in mind, there are also astronomical explanations which are not in conformance with the Rambam's explanations in H. Yesodei HaTorah. Kasher also points out that the author of the commentary also professes an unfamiliarity with certain astronomical Arabic terms which is hardly conceivable of the Rambam a man whose native tongue was Arabic and who was thoroughly immersed in the sciences of the day. Further, there are sections of what is understood to be the Rambam's actual commentary quoted by the Baal HaMaor (a commentary on the Rif by Zerakhia haLevi of Gerona) which are altogether absent from this commentary, which further strengthens the contention that it is a mistake to attribute it to the Rambam.

If Kasher's criticisms are to be deemed reasonable and the copyist's "Rabbeinu Moshe" ought not accordingly be conflated with the Rambam, we are then left with no substantive evidence of a pilgrimage having been made by the Rambam to the Temple Mount, as romantic as such a notion may be. There ought be an honest discourse surrounding the issue of whether it is halakhically acceptable today to ascend and I believe that the way this story is so often told is in order to make an emotional argument in its favor. Setting aside the legitimacy of such a tactic in and of itself, given that there is such a major question mark surrounding the authenticity of attributing this story to the Rambam I believe that those that favor ascent today would do well to set this story aside.