Really like the idea as there isn't really any good way to handle errors from the error stream. The only way known to me is something like this:

Get-Whatever 2>&1 |
  ForEach-Object {
    if ($_ -is [ErrorRecord]) {
      # handle error
    } else {
      $_
    }
  }

However, the break and continue seem weird to me because this is a scriptblock, not a loop.
Why not continue by default, and break with throw? That would be more intuitive to me

So, how does it work for distinguishing the error type and acting accordingly? Let's say there is a script in which a file has to be deleted. The Remove-Item command might end with an error when:

1. The file is not found: Well, that's good, right? We wanted the file gone and it is. The script can disregard this and move on.
2. Access to the file is denied: Not good! We wanted the file gone and it is still alive. The script must not move on.

You would presumably get $_ input into the scriptblock as the error record, so you could pick what to do based on the type of $_.Exception that it receives.

Just to call it out, PR #8205 is currently in the review process and should be in place in PowerShell 7 shortly (it's just waiting for PR #9825 to finish review and be merged so that automated scripts can be written against the debugger). It adds an ActionPreference.Break enumeration, which allows you to do this:

Stop-Process foo* -ErrorAction Break

That command will automatically enter the debugger on terminating or non-terminating error, and from that point you can check the call stack, step, continue execution, or terminate (quit) the script that is running.

Once that is merged, it will resolves some of the needs identified here (breaking into the debugger and/or checking the callstack on error).