We perform a time-dependent density functional theory (TDDFT) investigation for the optical properties of nanorod assemblies for different sizes (Ag-10, Ag-59, and Ag-139), interparticle distances, and orientations with a focus on the effect of symmetry breaking via an angle on plasmon coupling. For the model systems, the angle (theta) between the particles is varied between 0 and 180 degrees, where theta = 0 degrees and theta = 180 degrees correspond to symmetric side-by-side and end-to-end orientations of the nanorods, respectively. Our analysis reveals that for a sufficiently large interparticle distance (r > 0.7 nm), where the wave-function overlap between monomers is negligible, TDDFT results agree quite well with the predictions of the dipole-dipole interaction model for the intensity of the different modes of coupled plasmons. For smaller gap distances (0.4-0.5 nm), a charge-transfer plasmon (CTP) mode occurs for the symmetry broken case of the Ag-10 dimer. For the assemblies of larger nanorods, however, the CTP mode is predicted to be less pronounced, especially for the cases where the deviation from the end-to-end geometry is larger than 30 degrees. The orbital overlap and configuration-interaction analyses show that these results are related to the fact that the relative overlap strength between monomeric energy levels is significantly reduced for symmetry-broken orientations of larger nanorods.