You have to take an in depth look at yourself, really. Take a look at how each organization operates, and ask yourself if that's how you want to run with things. Don't fall for the glamorous, high-profile BS that everyone thinks of when looking at a career in special operations. Realize that there's going to be a bunch of bullshit as well, and that the glamorous moments aren't as bountiful as the everyday moments. Yeah, it's cool to go house tippin', but that's just a small part of the bigger picture. Look at that bigger picture, see if that's the picture you want to be in. Once you have a good sight picture, acquire your 25m target and move out.
I'm obviously neither SF nor a SEAL. However, I did have to consider some of the same things when deciding whether or not to cross over into EOD. The glamor of huge explosions vs. the waiting on calls.... having people say "Holy shit, you do this?" vice asking myself "Holy shit why do I do this?" after spending several hours in that damned bomb suit.... the little sliver of EOD that the public sees vs. the big picture, and I chose to be a part of it.
Lastly, realize that whatever path you follow, you're committing to more than just a job. It's not just another MOS in that branch of service. It's a way of life, like no other. Those you serve with will be your brothers, your family, just as close - if not closer - than your own blood kin. Look at the men you wish to call your brothers, and go with what is right. I will not advocate one over the other, as that choice is one that you alone must make. Do a gut check, a little soul searching, see where you want to be, and then go for it.
Sorry it ran a little long, I hope this helps some.