Fish stock assessment models often rely on size- or age-specific observations that are assumed to be statistically independent of each other. In reality, these observations are not raw observations, but rather they are estimates from a catch-standardization model or similar summary statistics based on observations from many fishing hauls and subsamples of the size and age composition of the data. Although aggregation mitigates the strong intra-haul correlation between sizes/ages that is usually found in haul-by-haul data, violations of the independence assumption can have a large impact on the results and specifically on reported confidence bounds. A state-space assessment model that allows for correlations between age groups within years in the observation model for catches and surveys is presented and applied to data on several North Sea fish stocks using various correlation structures. In all cases the independence assumption is rejected. Less fluctuating estimates of the fishing mortality is obtained due to a reduced process error. The improved model does not suffer from correlated residuals unlike the independent model, and the variance of forecasts is decreased.