The visual system is able to detect targets according to a variety of criteria, such as by categorical (letter vs digit) or featural attributes (color). These criteria are often used interchangeably in rapid serial visual presentation (RSVP) studies but little is known about how rapidly they are processed. The aim of this work was to compare the time course of attentional selection and memory encoding for different types of target criteria. We conducted two experiments where participants reported one or two targets (T1, T2) presented in lateral RSVP streams. Targets were marked either by being a singleton color (red letter among black letters), being categorically distinct (digits among letters) or non-singleton color (target color letter among heterogeneously colored letters). Using event related potential (ERPs) associated with attention and memory encoding (the N2pc and the P3 respectively), we compared the relative latency of these two processing stages for these three kinds of targets. In addition to these ERP measures, we obtained convergent behavioral measures for attention and memory encoding by presenting two targets in immediate sequence and comparing their relative accuracy and proportion of temporal order errors. Both behavioral and EEG measures revealed that singleton color targets were attended much more quickly than either non-singleton color or categorical targets, and there was very little difference between attention latencies to non-singleton color and categorical targets. There was however a difference in the speed of memory encoding for non-singleton color and category latencies in both behavioral and EEG measures, which shows that encoding latency differences do not always mirror attention latency differences.