Tracklogs
Virtually all handheld GPS units allow you to automatically record where you've been.
But to be most useful for GeoBiking, you need a unit that supports displaying multiple saved tracklogs. This eliminates many older GPS receivers and limits the usefulness of some new units as well! Magellan Explorist series supports 3-5 saved tracks. However only one is visible at at time. Most Garmin handhelds support viewing at least 10 tracks, the color units support 20. Lowrance iFinder series support up to 100 tracks (but limit tracks to 10,000 pts total).
The DeLorme PN-40/20 allows 10 saved tracks, but you can view only one at a time. However, you can copy the tracks to a draw layer and display them all as shown on the home page. It is the only unit I know of that will allow my entire trail network to be displayable on the receiver.
Big Sunlight-Readable Screen
You must be able to see the display in varying light conditions from bright sun to shadows. This is a more problematic issue with color displays. The larger the display, the more map data you can show and the quicker it can be digested. Some displays such as the Bushnell Onix play with polarization that may reduce glare to the naked eye but make it unuseable with polarized sunglasses.
In any case, I suggest putting a vinyl screen protector over the display. It is cheap insurance to maintain screen clarity over the long haul.
Color vs Black & White
Color displays make it much easier to get information quicker. It is easier & faster to differentiate the displayed information. This really helps to allow quick glances enroute. On color units, you can change the color of the tracklogs to help make them stick out from the background data. The best sunlight readable color displays I've seen are newer Garmin units and the DeLorme PN-20. The Magellan color displays have issues with sunlight. For Lowrance, the Hunt color has less resolution than the Expedition color, but I think it is more sunlight viewable.
Ease of Data Entry If you are following tracks and not creating your own data, the ease of entering information is not as important. Most older units & Lowrance units require you to go up/down thru numbers & alphabet to make entries. Units with on screen keyboards help make this a little easier/intuitive.
Processor Speed
Some units react a lot slower to input than others. Noticeable among these are the Lowrance "Go" models. The DeLorme PN series and Lowrance iFinder series use dual porcessor's to help speed map panning etc. This makes it easier to operate, but harder on batteries.
Mapping vs non-Mapping
A mapping GPS is not a requirement for GeoBiking. It does make it nicer for mapping trips though. I don't know of any recent color GPS that doesn't do maps. Non-Mapping B/W units may actually have an advantage over their mapping counterparts in reduced clutter. I've missed trail connections on B/W mapping units because the trails just didn't stick out from the map.
Computer Interface
The GPS must of course have a computer interface or other means to transfer track & waypoint information. Most newer units have USB interfaces. Older units may use serial interfaces that are slower and are not normally found on newer computers. The Lowrance iFinder series transfers data ONLY via removable memory card - something of a PITA.
Memory
About any unit out there will allow you to record/use 500 waypoints and 20 routes. These are almost always stored in internal memory. Map memory can be internal or external. For GeoBiking, almost any unit that allows use of maps will be able to handle more than you can bike in a few days.
Reception Ability
You don't normally need a high sensitivity receiver to follow tracks. It can however, greatly improve the quality of the trails you record. It becomes more of an issue when riding in thick forests.
Regular batteries vs packs
The Magellan Explorist 400, 500 & 600 and Garmin Rino color units use a battery pack for power. It allows more energy in a given space but requires you to have extra packs available. Both the Magellan & Garmin units provide extra cost adapters to use regular batteries. The battery life on the Explorist are greatly reduced with 3 AAA batteries. The Rinos allow you to use 3 AA batteries but limits radio power to 2 Watts instead of 5. I much prefer the ability to use AA batteries, they're available anywhere and rechargeable is still an option.
Touch Screen
The Triton 1500 & 2000 from Magellan and the new Garmin Oregon units utilize touch screens. They are nice for quickly entering data (e.g. waypoint/track names. How useful they are with gloves, muddy hands etc is yet to be seen.