Some time ago, after some X upgrade, I observed that when I switched from X to a normal tty I couldn't switch back to X without losing the session. Today I decided it is worth the time to investigate.
So, reproduce and look for a core. Found a fresh Xorg.core under /. Next gdb `which Xorg` Xorg.core shows that it was an assertion failure in Xorg itself, and not in one of the libraries it links against. Hence env WITH_DEBUG=YES pu-install x11-servers/xorg-server (i.e., reinstall the port x11-servers/xorg-server building it with WITH_DEBUG set, to get an unstripped version (pu; steps)). Another reproduction of the error, another gdb, and there I was: xf86_reload_cursors did a dixLookupPrivate on xf86CursorScreenKey despite not xf86CursorScreenKey->initialized. Given the description of xf86_reload_cursors ("Reloads cursor images as needed, then adjusts cursor positions"), I decided that the worst I could lose by exiting this function (fortunately returning void) early was the cursor image—and that this was still better than crashing; also the return on the lookup returning a false value (and the various other conditional returns) gave me hope that returning here early couldn't do too much harm.
Hence I dropped the the following patch in my local patches (based on the EXTRA_PATCHES mechanism).
--- hw/xfree86/modes/xf86Cursors.c.orig 2015-06-13 21:18:49.729903000 +0200 +++ hw/xfree86/modes/xf86Cursors.c 2015-06-13 21:23:35.604525000 +0200 @@ -625,6 +625,8 @@ */ if (!screen || !inputInfo.pointer) return; + if (!(xf86CursorScreenKey->initialized)) + return; cursor_screen_priv = dixLookupPrivate(&screen->devPrivates, xf86CursorScreenKey); /* return if HW cursor is inactive, to avoid displaying two cursors */