Lot of good suggestions. If I was C/Maj I would get with some people local who are knowledgeable and discuss this with them.
One aspect is; so you bought the wrong first rifle. Nothing wrong with owning more than 1 rifle. Unless you paid way to much for it, it will eventually be worth what you paid for it. (maybe a long wait).
There is a fit issue, does the rifle fit your body, same goes with pistols.
Recoil issue, it has been mentioned. Felt recoil can be a big issue. The way a M4 is shoulder and fired does not completely translate over to shooting a bigger bore rifle.
Target shooting, for fun anything works. If you get competitive, then it becomes very specialized.
Hunting, depends on game and area.
Lots of choices. Over the year my taste have changed or been redefined. I like about anything that goes bang. My oldest shooter is a 1818 Springfield.
I think it helps, if you have friends who know about what you have. They can help. You maybe able to try some rifles out before you commit to buying one and the stuff you will mount on it. Glass (scopes) will not make you a better shooter. If you can not shoot with iron sights, you won't be able to shoot better with a scope. (better means consistency).
Reloading is a necessity if you get into completive shooting. If you don't shoot much, it is probably not worth it, unless you want to know more.
BTW, if you do decide on a rifle, Let us know.
This is my totally cool tactical CQC battle rifle shown w/o bayonet: