I see two factors playing into this:
1. The spotlight/Hollywood 'special' effect of SEALs (as compared to other SOF) on recruit expectations. In the 70s and 80s you had Rambo, Commando, A-Team, Delta Force, and a slew of other cinema that highlighted Army SOF. From the 90s on though, with the exception of Black Hawk Down, SEALs have mostly dominated the big budget SOF TV/film landscape, and definitely the gaming circuit. Due to this a larger percentage of SEAL Challenge candidates might be the types who are sold on the 'shiny special' status of the job as compared with, say, your Option 40 or 1T2X1 recruits.
2. The Navy, unlike the Army or Marine Corps, doesn't have the 'safety net' of conventional combat arms positions for unsuccessful SOF candidates to pivot towards after getting dropped. Most off-the-street SOF applicants probably aren't joining with the expectation of settling into something unrelated to SOF or combat arms, which adds to the stakes of failing for non-Army/Marine Corps candidates.
I'd be interested in hearing from
@amlove21 or anyone else from the AFSPECWAR community on if there aren't similar issues with candidates despising the Air Force after washing out of selection.
They sure can't seem to get enough of their seamen.