Repository navigation
Implements the user errors feature for RCV2 - #90
Joseph Calev (jscalev) wants to merge 7 commits into
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
Error propagation can skip command execution, change handler exit behavior, and produce conflicting terminal statuses.
5 open issues
Allow script retry when initial status upload fails · New Honor deployment-failure setting in script error handling · New Treat already-processed sentinel as normal completion · New Avoid conflicting fallback results after reported script failures · New Clean up runtime settings even when script directory fails · New
What changed in this PR
This PR adds classified errors to RunCommand V2 so status reporting can distinguish customer or VM problems from handler and Azure infrastructure failures.
Changes:
- Propagates clarification codes through execution, downloads, settings, services, and lifecycle operations.
- Includes error-clarification substatuses in managed and immediate status reports.
- Adds failure-injection tests, stronger code assertions, and Windows-compatible testing helpers.
| File | Description |
|---|---|
| README.md | Documents clarification codes, reporting, and testing. |
| pkg/versionutil/version.go | Classifies invalid service definitions. |
| pkg/versionutil/version_test.go | Asserts version-parsing clarification codes. |
| pkg/versionutil/clarification_test.go | Tests invalid-definition classification. |
| pkg/urlutil/urlerrorhelper.go | Preserves error causes during URL redaction. |
| pkg/urlutil/urlerrorhelper_test.go | Asserts redaction clarification codes. |
| pkg/urlutil/clarification_test.go | Tests redaction and classification preservation. |
| pkg/systemd/manager.go | Classifies systemd operation failures. |
| pkg/systemd/helper.go | Classifies configuration-directory failures. |
| pkg/systemd/clarification_test.go | Injects systemd and filesystem failures. |
| pkg/statusreporter/statusreporter.go | Classifies request failures and unsuccessful HTTP responses. |
| pkg/statusreporter/guestinformationserviceclient.go | Classifies empty status uploads. |
| pkg/statusreporter/clarification_test.go | Tests status-upload failure classifications. |
| pkg/servicehandler/servicehandler.go | Preserves classifications across service operations. |
| pkg/servicehandler/servicehandler_test.go | Asserts service failure codes. |
| pkg/servicehandler/clarification_test.go | Tests service classification and wrapped missing files. |
| pkg/seqnumutil/seqnum.go | Classifies sequence errors and fixes filename collection. |
| pkg/seqnumutil/clarification_test.go | Tests sequence discovery and persistence failures. |
| pkg/preprocess/file.go | Classifies text-file inspection failures. |
| pkg/preprocess/file_test.go | Asserts file-open classification. |
| pkg/preprocess/clarification_test.go | Tests injected file failures and propagation. |
| pkg/linuxutils/linuxutils.go | Adds classified cleanup and latest-sequence exclusions. |
| pkg/linuxutils/clarification_test.go | Tests cleanup failures and preserved files. |
| pkg/errorutil/errors.go | Adds classification and wrapped-error lookup helpers. |
| pkg/download/url.go | Classifies invalid download requests. |
| pkg/download/url_test.go | Asserts invalid-URL classification. |
| pkg/download/save.go | Classifies destination and download failures. |
| pkg/download/save_test.go | Asserts destination-open classification. |
| pkg/download/retry.go | Preserves latest retry classifications and prior causes. |
| pkg/download/retry_test.go | Asserts retry failure codes. |
| pkg/download/downloader.go | Classifies transport and HTTP failures. |
| pkg/download/downloader_test.go | Tests download failure classifications. |
| pkg/download/blobwithmsitoken.go | Classifies managed-identity token failures. |
| pkg/download/blob.go | Classifies blob operations and fixes error-status handling. |
| pkg/download/blob_test.go | Adds classification assertions and mocked network failure. |
| pkg/blobutil/url.go | Classifies blob URL validation failures. |
| pkg/blobutil/url_test.go | Asserts URL-specific clarification codes. |
| pkg/blobutil/clarification_test.go | Tests blob URL classification cases. |
| internal/types/status.go | Adds serialized error-clarification substatuses. |
| internal/types/runcommandmetadata.go | Carries clarification errors in command metadata. |
| internal/types/instanceview.go | Carries nonserialized instance-view clarification errors. |
| internal/types/commands.go | Defines the already-processed sentinel. |
| internal/telemetry/telemetry.go | Classifies telemetry write, publish, and close failures. |
| internal/telemetry/clarification_test.go | Tests telemetry failures and error precedence. |
| internal/status/status/immediateGoalStatesInTerminalStatusFile.status | Adds terminal-status JSON data. |
| internal/status/status.go | Propagates clarifications and classifies persistence failures. |
| internal/status/status_test.go | Verifies persisted clarification substatuses. |
| internal/status/immediatestatus.go | Handles substatuses and classifies reporting failures. |
| internal/status/immediatestatus_test.go | Isolates persistence and asserts reporting codes. |
| internal/settings/settings.go | Classifies JSON errors and validates settings types. |
| internal/settings/settings_test.go | Checks raw JSON syntax errors remain unclassified. |
| internal/settings/clarification_test.go | Tests settings validation classifications. |
| internal/service/serviceinstall.go | Classifies unsupported systemd and executable-permission failures. |
| internal/service/clarification_test.go | Tests service-registration failure codes. |
| internal/requesthelper/retry.go | Detects retryable errors through wrappers. |
| internal/requesthelper/retry_test.go | Asserts retry clarification codes. |
| internal/requesthelper/request.go | Classifies request creation, transport, and response failures. |
| internal/requesthelper/request_test.go | Asserts request failure codes. |
| internal/requesthelper/clarification_test.go | Tests classified failures and retry preservation. |
| internal/pid/pid.go | Classifies process-record and signaling failures. |
| internal/pid/pid_windows.go | Adds unsupported Windows process-group signaling. |
| internal/pid/pid_windows_test.go | Tests the unsupported-operation sentinel. |
| internal/pid/pid_test.go | Asserts process-start lookup classification. |
| internal/pid/pid_linux.go | Supplies Linux process-group signaling. |
| internal/pid/clarification_test.go | Tests process-record and signaling classifications. |
| internal/observer/notifier.go | Classifies observer notification failures. |
| internal/observer/clarification_test.go | Tests notification classification preservation. |
| internal/instanceview/instanceview.go | Passes clarification errors into status reporting. |
| internal/immediateruncommand/immediateruncommand.go | Validates goal states and reports classified failures. |
| internal/immediateruncommand/clarification_test.go | Tests incomplete states and concurrency-limit codes. |
| internal/immediatecmds/immediatecmds.go | Returns the actual service-uninstall error. |
| internal/hostgacommunicator/vmsettings.go | Classifies request and certificate-check failures. |
| internal/hostgacommunicator/vmsettings_test.go | Asserts response-parsing classification. |
| internal/hostgacommunicator/hostgacommunicator.go | Classifies response failures instead of panicking. |
| internal/hostgacommunicator/clarification_test.go | Tests host communication and certificate failures. |
| internal/handlersettings/types.go | Classifies inconsistent artifact specifications. |
| internal/handlersettings/handlersettingscommon.go | Preserves settings and decryption error causes. |
| internal/handlersettings/handlersettings.go | Classifies missing script sources. |
| internal/handlersettings/handlerenv.go | Classifies environment discovery and parsing failures. |
| internal/goalstate/goalstatefromvmsettings.go | Classifies goal-state retrieval failures. |
| internal/goalstate/goalstatefromvmsettings_test.go | Asserts retrieval failure classification. |
| internal/goalstate/goalstate.go | Propagates immediate-command failures and clarification status. |
| internal/goalstate/goalstate_test.go | Verifies skipped-state clarification substatuses. |
| internal/goalstate/clarification_test.go | Tests goal-state failures and status propagation. |
| internal/files/logtail.go | Classifies output-file access failures. |
| internal/files/logtail_test.go | Asserts tail-file open classification. |
| internal/files/files.go | Classifies script processing and handles write permissions. |
| internal/files/files_test.go | Asserts script and URL failure codes. |
| internal/extensionpolicysettingsrc/types.go | Classifies policy-format and script-type violations. |
| internal/extensionpolicysettingsrc/types_test.go | Asserts policy validation codes. |
| internal/extensionpolicysettingsrc/extensionpolicysettingsrc.go | Classifies policy loading and enforcement failures. |
| internal/extensionpolicysettingsrc/extensionpolicysettingsrc_test.go | Asserts policy enforcement classifications. |
| internal/exec/exec_windows.go | Defines Windows RunAs file-opening flags. |
| internal/exec/exec_test.go | Asserts execution and output-file failure codes. |
| internal/exec/exec_linux.go | Defines Linux no-follow file-opening protection. |
| internal/commandProcessor/commandProcessor.go | Reports clarified command and precheck failures. |
| internal/commandProcessor/commandProcessor_test.go | Tests classified prechecks and sequence failures. |
| internal/cleanup/cleanup.go | Uses shared cleanup helpers and classified logging. |
| internal/cleanup/clarification_test.go | Tests cleanup classification and propagation. |
| cmd/main/main.go | Separates command parsing and classifies invalid arguments. |
| cmd/main/clarification_test.go | Tests command-line clarification codes. |
Review effort: Balanced
birukyitbarek
left a comment
There was a problem hiding this comment.
Leverage copilot to do these reviews, if more context is needed happy to provide.
Also some of the comments were made 3 days ago but weren't published so if you made iteration changes since some of the comments might be out of line but should be easy to understand what code block they are referring to.
| var goalStateKeys []types.GoalStateKey | ||
| for _, s := range goalStates { | ||
| for _, setting := range s.Settings { | ||
| if err := validateGoalStateSettings(setting); err != nil { |
There was a problem hiding this comment.
if one goal state has a missing ExtensionState, ExtensionName or SeqNo, validateGoalStateSettings rejects the whole batch. The ETag doesn't advance, so the service retries every 5 seconds forever and the valid goal states never run. A missing ExtensionState used to be allowed, so the check at :99 is now dead code. Skip or report only the bad entry.
There was a problem hiding this comment.
Same comment for line 217 below.
| ctx.Log("error", fmt.Sprintf("error when trying to execute goal state: %v", e)) | ||
| var executionErr *ImmediateCommandExecutionError | ||
| if errors.As(e, &executionErr) { | ||
| return executionErr.InstanceView.ExitCode, errors.Wrap(e, "error when trying to execute goal state") |
There was a problem hiding this comment.
this returns InstanceView.ExitCode, which is 0 when the script succeeded but reporting failed. The caller gets exit code 0 together with an error, so use a non-zero code there.
| if err != nil && runErr == nil { | ||
| runErr, exitCode = err, constants.ExitCode_BlobCreateOrReplaceFailed | ||
| } | ||
| errorFilePosition, err = appendToBlob(stderrF, errorBlobSASRef, errorBlobAppendClient, errorFilePosition, ctx) |
There was a problem hiding this comment.
if appending output to the blob fails after a successful script, the run now fails with ExitCode_BlobCreateOrReplaceFailed. Customers will see successful scripts reported as failed.
|
|
||
| scriptPath := cmd | ||
|
|
||
| commandArgs, err := SetEnvironmentVariables(cfg) |
There was a problem hiding this comment.
a Setenv failure now stops the script from running; before, the script still ran. That may be fine, but it's a behavior change worth calling out.
| @@ -0,0 +1,316 @@ | |||
| package errorutil | |||
|
|
|||
There was a problem hiding this comment.
General comment on pattern:
Positive codes are user errors and negative codes are system errors. But that doesn't appear to be followed consistently.
Examples:
Similar OS failures get opposite classes:
• Deleting the data directory (:12) is a system error (-50), but creating it (:292) is a user error (5400).
• Opening stdout (:13) is a system error (-51), but closing it (:277) is a user error (5013).
Files the handler writes itself are classified as user errors:
• Os_InvalidTerminalGoalStatesJson (:288, 5208)
• Os_InvalidStoredSequenceNumber (:239, 4204)
• Os_InvalidProcessRecordFormat (:214, 3654)
Yet Internal_invalidMrseqFileSequenceNumber (:92, -5412) is a system error, so the same corrupt-state condition gets both classes.
Files written by the platform or guest agent are classified as customer errors:
• CustomerInput_missingCertificateThumbprint and CustomerInput_invalidProtectedSettingsBase64 (:168–169, used at handlersettingscommon.go:106)
• Os_InvalidHandlerEnvironmentJson (:163)
• CustomerInput_unexpectedRuntimeSettingsCount (handlersettingscommon.go:93)
Meanwhile, invalid JSON in the same file two lines up (handlersettingscommon.go:90) is Internal_badConfig, a system error
There was a problem hiding this comment.
I made this comment 3 days ago (but forgot to publish) and if you updated this file since, some of the line numbers might be out of whack.
| statusItem, err := status.GetSingleStatusItem(ctx, statusType, c, msg, metadata.ExtName) | ||
| if statusType != types.StatusTransitioning { | ||
| var view types.RunCommandInstanceView | ||
| if err := json.Unmarshal([]byte(msg), &view); err != nil { |
There was a problem hiding this comment.
if json.Unmarshal fails on a finished command's result, the code returns before notifier.Notify. The final status is never sent to the host agent, though it always was before.
| cfg, err := handlersettings.GetHandlerSettings(hEnv.HandlerEnvironment.ConfigFolder, extensionName, seqNum, ctx) | ||
| // Fail the extension status and handler invocation only when deployment failure is enabled and the command has a non-zero failure exit code. | ||
| cfg, err := readHandlerSettings(hEnv.HandlerEnvironment.ConfigFolder, extensionName, seqNum, ctx) | ||
| if err == nil && cfg.PublicSettings.TreatFailureAsDeploymentFailure && cmd.FailExitCode != 0 { |
There was a problem hiding this comment.
This block now returns an error, so main.go:33 exits with FailExitCode, which changes the handler's exit code. On main it effectively returned nil.
• A GetHandlerSettings failure is still silently ignored.
• The ClarificationError substatus has status "error" even when the top-level status is Success.
| // Report the output streams to blobs | ||
| outputFilePosition, err = appendToBlob(stdoutF, outputBlobSASRef, outputBlobAppendClient, outputFilePosition, ctx) | ||
| errorFilePosition, err = appendToBlob(stderrF, errorBlobSASRef, errorBlobAppendClient, errorFilePosition, ctx) | ||
| outputFilePosition = appendOutput(stdoutF, outputBlobSASRef, outputBlobAppendClient, outputFilePosition) |
There was a problem hiding this comment.
if appending output to the blob fails after a successful script, the run now fails, so customers see successful scripts reported as failed.
| exitCode = code | ||
| if signaled { | ||
| ctx.Log("message", "Timeout:"+err.Error()) | ||
| return exitCode, errorutil.EnsureClarification(errorutil.CommandExecution_interruptedByVmShutdown, fmt.Errorf("command terminated with exit status=%d", exitCode)) |
There was a problem hiding this comment.
nit: every process killed by a signal is reported as interruptedByVmShutdown. Timeouts, the out-of-memory killer and the user's own kill would all be misreported.
| func ValidateScriptTypeAgainstPolicy(ctx *log.Context, scriptType handlersettings.ScriptType, allowedScriptTypesString string) error { | ||
| allowedScriptTypes, _ := StringToAllowedScriptTypeFlag(allowedScriptTypesString) | ||
| allowedScriptTypes, parseErr := StringToAllowedScriptTypeFlag(allowedScriptTypesString) | ||
| if parseErr != nil && allowedScriptTypesString != "" { |
There was a problem hiding this comment.
the message says the script type isn't allowed, but the real problem is a malformed policy file.


This implements the user errors project for RCV2 to better differentiate user from customer errors.