A very helpful and widely recommended paper (and accompanying book) in this domain is: Digital Communication Systems Using Matlab And Simulink

– particularly the editions that include the MATLAB/Simulink companion or the dedicated "Digital Communications Using MATLAB and Simulink" by Sklar and Ray. A very helpful and widely recommended paper (and