Failure and Frustration
I've been playing Alien Isolation a lot recently. It's HARD, its fail states suck, and its save system is punishing, but the tension that fear of punishment creates is thrilling. It's a great example of how something typically done by the numbers can be used deliberately to create effect.
Monument Valley is almost the polar opposite. A "failure" in Monument Valley (encountering one of the black birds) merely pushes you backwards a little, and forces you to wait for your next opportunity or rethink your approach. It's as little resistance as you could possibly have without removing obstacles entirely.
In the original Mallow Drop, when you died / failed a puzzle, the level reset and you tried again. This was about as good as I could code at the time, but it was fairly easy to see that it sucked.
But why? What makes repeating something in a game so frustrating?
Is it frustration at being defeated, or at the time wasted redoing the things we got right?
If you get an answer wrong on a test, would you rather redo that one question or redo the whole test?
The things that seemed like small victories no longer have that thrill if there isn't at least an opportunity to redo them better the second time around.
There's also no tension of what failure might mean the second time around. You KNOW what failure means because that's the state you're currently in. The threat of failure is doing exactly what you're doing now. This is why dieing in Alien Isolation is a rush the first time it happens in an area, but repeated attempts at the same area become more frustrating and boring than tense and interesting.
So, what are some of the OTHER ways games deal with failure, rather than a hard reset?
Pretty much the core of every RPG system. Makes the game relatively easier based on time spent playing. A player who races ahead will find the game harder. One progresses slowly will find that progression easier.
The problem with this is that it often encourages grinding or absenteeism (not engaging with the story), in order to make later progress easier. I absolutely hate this, as it encourages playing the game more, so that you don't have to play the game.
The spelunky approach. If the play area is different every time, then failure is a different challenge/punishment every time. It encourages learning to adapt over memorisation.
Can sometimes create an easier area, removing the challenge. Progression can feel the result of the generator, not the player.
Particularly in esports games, there has to be a cutoff point where a player has either won or lost, and the play has completely ended.
There are two main ways this happens: By points total, or point-difference
Point difference theoretically allows for a game to continue indefinitely, as long as both sides are evenly matched, so it becomes important to make changes to the area/player abilities to allow for one to get an advantage as the game progresses
Allowing progression in spite of failure
Professor Layton takes a fantastic approach to failure. Any puzzle you deem too hard can simply be put in the "too hard basket" to be revisited later. This is a brilliant move, as it demonstrates an understanding of how people often interact with puzzles (going away, and coming back only to have the solution staring them in the face) while also allowing story progression to continue in spite of that leaving.
Broken Age also uses this as it's central design philosophy. Any time you're stuck on Shay or Vella's storyline, you can jump to the other and still progress the story.
Reducing the impact of failure
This is the approach I've taken with the rebuild of Mallow Drop, and is also taken with many great puzzle games. A "wrong" move or series of moves should be able to be undone to allow the player to rewind the game to the last known "good" state. This does not necessarily mean solvable, but where the player BELIEVES they went wrong.
It's one of the things I love most about Fez. As a platformer that's more about exploration than it is about speed and skill, Fez's deaths from falling are handled generously. You simply get put back on the platform you fell from. It takes almost no time, and it allows players to explore various options without fear of redoing entire sections of the game. It's also notably used in the Prince of Persia series, but only up to a point, at which the player is sent back to the start of the area.
It identifies a "wrong" state, either explicitly by killing the player and resetting, or implicitly by not having further moves available, but not necessarily the decisions that lead to that state. It doesn't show the solution through the act of showing the fact the player hasn't reached it.
While it reduces the amount of redoing correct moves (important for a puzzle game), it reduces the feeling of freedom somewhat. This is fine for linear games with a clear progression like Mallow Drop or Puzzle Retreat, but it wouldn't work in a game like Alien Isolation. It'd kill the mood entirely, instead of just you.