Do you want to design the whole thing from scratch, or use a pre-existing design? The 6502 is a pretty simple chip, so most basic designs look very much alike, and the real points of deviation will be your bank-switching scheme and maybe your disk or flash device interface, maybe your video if you have it, but even that’s pretty vanilla if you use a 6845 or such. There are several nice designs on 6502.org, including one with memory management and its own multitasking OS. Can’t remember the name off the top of my head.
We all love the 6502. But the 6809 is probably the pinnacle of the 8-bitters, and has a beautiful fairly orthogonal instruction set, is as easy to interface as the 6502, and is it’s natural successor, within the 8-bit constraint. If programming a 6502 can be likened to breathing through a straw, then programming the 6809 is like a deep breath of fresh air, almost like programming a PDP-11 or 68000. It also runs perhaps the best 8-bit OS of the period, Microware’s OS/9, now reincarnated and actively developed as NitrOS/9.
For memory management, you can get as simple as a latch for mapping banks of RAM into a bankable window, or you could use a CPLD for flexible mapping, but there exists a family of special TTL chip, the 74LS610-13 chips, that aggregate a lot of function, including an addressable register bank and decoder in a single chip. The '10 is 3-state and latching. There are examples of its use online, including 6502.org.
There are at least a couple of unix-like systems that can compile on 6502. LUnix (in assembly) and con-tiki (in C, via cc65) off the top of my head.
If you’re interested in other OS’s besides a unix workalike, there’s a nice portable, well written CP/M workalike called DOS/65. The 6502 source is here.
If you are interesting in recreating a highly configurable and hackable basic machine from the late 70’s-early 80’s, there’s a project underway to create high-fidelity reproductions of a number of Ohio Scientific (OSI) PCB’s, as well as new boards for added functionality. These are enough to create a very configurable system including video, disk, RAM/ROM. 20 address-bus lines allow for bank switching across the system. For example, one CPU board allows switching between 6502, 6800, or Z80, and runs CP/M or 6502-based OS’s. Another adds an intersil 6100 under control of the system CPU to provide a PDP/8e compatible system. If you’re interested, you can check:
http://osiweb.org/#Links:
http://osiweb.org/osiforum/viewtopic.php?f=3&t=322
http://osiweb.org/osiforum/viewtopic.php?f=3&t=332
http://osiweb.org/osiforum/viewtopic.php?f=3&t=317
Dave