Skip to content

Implements the user errors feature for RCV2 - #90

Open
Joseph Calev (jscalev) wants to merge 7 commits into
mainfrom
dev/jcalev/ImplementUserErrors
Open

Joseph Calev (jscalev) wants to merge 7 commits into
mainfrom
dev/jcalev/ImplementUserErrors

Conversation

@jscalev

Copy link
Copy Markdown
Contributor

This implements the user errors project for RCV2 to better differentiate user from customer errors.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Error propagation can skip command execution, change handler exit behavior, and produce conflicting terminal statuses.

5 open issues
What changed in this PR

This PR adds classified errors to RunCommand V2 so status reporting can distinguish customer or VM problems from handler and Azure infrastructure failures.

Changes:

  • Propagates clarification codes through execution, downloads, settings, services, and lifecycle operations.
  • Includes error-clarification substatuses in managed and immediate status reports.
  • Adds failure-injection tests, stronger code assertions, and Windows-compatible testing helpers.
File Description
README.md Documents clarification codes, reporting, and testing.
pkg/​versionutil/​version.go Classifies invalid service definitions.
pkg/​versionutil/​version_test.go Asserts version-parsing clarification codes.
pkg/​versionutil/​clarification_test.go Tests invalid-definition classification.
pkg/​urlutil/​urlerrorhelper.go Preserves error causes during URL redaction.
pkg/​urlutil/​urlerrorhelper_test.go Asserts redaction clarification codes.
pkg/​urlutil/​clarification_test.go Tests redaction and classification preservation.
pkg/​systemd/​manager.go Classifies systemd operation failures.
pkg/​systemd/​helper.go Classifies configuration-directory failures.
pkg/​systemd/​clarification_test.go Injects systemd and filesystem failures.
pkg/​statusreporter/​statusreporter.go Classifies request failures and unsuccessful HTTP responses.
pkg/​statusreporter/​guestinformationserviceclient.go Classifies empty status uploads.
pkg/​statusreporter/​clarification_test.go Tests status-upload failure classifications.
pkg/​servicehandler/​servicehandler.go Preserves classifications across service operations.
pkg/​servicehandler/​servicehandler_test.go Asserts service failure codes.
pkg/​servicehandler/​clarification_test.go Tests service classification and wrapped missing files.
pkg/​seqnumutil/​seqnum.go Classifies sequence errors and fixes filename collection.
pkg/​seqnumutil/​clarification_test.go Tests sequence discovery and persistence failures.
pkg/​preprocess/​file.go Classifies text-file inspection failures.
pkg/​preprocess/​file_test.go Asserts file-open classification.
pkg/​preprocess/​clarification_test.go Tests injected file failures and propagation.
pkg/​linuxutils/​linuxutils.go Adds classified cleanup and latest-sequence exclusions.
pkg/​linuxutils/​clarification_test.go Tests cleanup failures and preserved files.
pkg/​errorutil/​errors.go Adds classification and wrapped-error lookup helpers.
pkg/​download/​url.go Classifies invalid download requests.
pkg/​download/​url_test.go Asserts invalid-URL classification.
pkg/​download/​save.go Classifies destination and download failures.
pkg/​download/​save_test.go Asserts destination-open classification.
pkg/​download/​retry.go Preserves latest retry classifications and prior causes.
pkg/​download/​retry_test.go Asserts retry failure codes.
pkg/​download/​downloader.go Classifies transport and HTTP failures.
pkg/​download/​downloader_test.go Tests download failure classifications.
pkg/​download/​blobwithmsitoken.go Classifies managed-identity token failures.
pkg/​download/​blob.go Classifies blob operations and fixes error-status handling.
pkg/​download/​blob_test.go Adds classification assertions and mocked network failure.
pkg/​blobutil/​url.go Classifies blob URL validation failures.
pkg/​blobutil/​url_test.go Asserts URL-specific clarification codes.
pkg/​blobutil/​clarification_test.go Tests blob URL classification cases.
internal/​types/​status.go Adds serialized error-clarification substatuses.
internal/​types/​runcommandmetadata.go Carries clarification errors in command metadata.
internal/​types/​instanceview.go Carries nonserialized instance-view clarification errors.
internal/​types/​commands.go Defines the already-processed sentinel.
internal/​telemetry/​telemetry.go Classifies telemetry write, publish, and close failures.
internal/​telemetry/​clarification_test.go Tests telemetry failures and error precedence.
internal/​status/​status/​immediateGoalStatesInTerminalStatusFile.status Adds terminal-status JSON data.
internal/​status/​status.go Propagates clarifications and classifies persistence failures.
internal/​status/​status_test.go Verifies persisted clarification substatuses.
internal/​status/​immediatestatus.go Handles substatuses and classifies reporting failures.
internal/​status/​immediatestatus_test.go Isolates persistence and asserts reporting codes.
internal/​settings/​settings.go Classifies JSON errors and validates settings types.
internal/​settings/​settings_test.go Checks raw JSON syntax errors remain unclassified.
internal/​settings/​clarification_test.go Tests settings validation classifications.
internal/​service/​serviceinstall.go Classifies unsupported systemd and executable-permission failures.
internal/​service/​clarification_test.go Tests service-registration failure codes.
internal/​requesthelper/​retry.go Detects retryable errors through wrappers.
internal/​requesthelper/​retry_test.go Asserts retry clarification codes.
internal/​requesthelper/​request.go Classifies request creation, transport, and response failures.
internal/​requesthelper/​request_test.go Asserts request failure codes.
internal/​requesthelper/​clarification_test.go Tests classified failures and retry preservation.
internal/​pid/​pid.go Classifies process-record and signaling failures.
internal/​pid/​pid_windows.go Adds unsupported Windows process-group signaling.
internal/​pid/​pid_windows_test.go Tests the unsupported-operation sentinel.
internal/​pid/​pid_test.go Asserts process-start lookup classification.
internal/​pid/​pid_linux.go Supplies Linux process-group signaling.
internal/​pid/​clarification_test.go Tests process-record and signaling classifications.
internal/​observer/​notifier.go Classifies observer notification failures.
internal/​observer/​clarification_test.go Tests notification classification preservation.
internal/​instanceview/​instanceview.go Passes clarification errors into status reporting.
internal/​immediateruncommand/​immediateruncommand.go Validates goal states and reports classified failures.
internal/​immediateruncommand/​clarification_test.go Tests incomplete states and concurrency-limit codes.
internal/​immediatecmds/​immediatecmds.go Returns the actual service-uninstall error.
internal/​hostgacommunicator/​vmsettings.go Classifies request and certificate-check failures.
internal/​hostgacommunicator/​vmsettings_test.go Asserts response-parsing classification.
internal/​hostgacommunicator/​hostgacommunicator.go Classifies response failures instead of panicking.
internal/​hostgacommunicator/​clarification_test.go Tests host communication and certificate failures.
internal/​handlersettings/​types.go Classifies inconsistent artifact specifications.
internal/​handlersettings/​handlersettingscommon.go Preserves settings and decryption error causes.
internal/​handlersettings/​handlersettings.go Classifies missing script sources.
internal/​handlersettings/​handlerenv.go Classifies environment discovery and parsing failures.
internal/​goalstate/​goalstatefromvmsettings.go Classifies goal-state retrieval failures.
internal/​goalstate/​goalstatefromvmsettings_test.go Asserts retrieval failure classification.
internal/​goalstate/​goalstate.go Propagates immediate-command failures and clarification status.
internal/​goalstate/​goalstate_test.go Verifies skipped-state clarification substatuses.
internal/​goalstate/​clarification_test.go Tests goal-state failures and status propagation.
internal/​files/​logtail.go Classifies output-file access failures.
internal/​files/​logtail_test.go Asserts tail-file open classification.
internal/​files/​files.go Classifies script processing and handles write permissions.
internal/​files/​files_test.go Asserts script and URL failure codes.
internal/​extensionpolicysettingsrc/​types.go Classifies policy-format and script-type violations.
internal/​extensionpolicysettingsrc/​types_test.go Asserts policy validation codes.
internal/​extensionpolicysettingsrc/​extensionpolicysettingsrc.go Classifies policy loading and enforcement failures.
internal/​extensionpolicysettingsrc/​extensionpolicysettingsrc_test.go Asserts policy enforcement classifications.
internal/​exec/​exec_windows.go Defines Windows RunAs file-opening flags.
internal/​exec/​exec_test.go Asserts execution and output-file failure codes.
internal/​exec/​exec_linux.go Defines Linux no-follow file-opening protection.
internal/​commandProcessor/​commandProcessor.go Reports clarified command and precheck failures.
internal/​commandProcessor/​commandProcessor_test.go Tests classified prechecks and sequence failures.
internal/​cleanup/​cleanup.go Uses shared cleanup helpers and classified logging.
internal/​cleanup/​clarification_test.go Tests cleanup classification and propagation.
cmd/​main/​main.go Separates command parsing and classifies invalid arguments.
cmd/​main/​clarification_test.go Tests command-line clarification codes.

Review effort: Balanced

Comment thread internal/goalstate/goalstate.go Resolved
Comment thread pkg/linuxutils/linuxutils.go Resolved
Comment thread internal/cleanup/cleanup.go Resolved
Comment thread pkg/errorutil/errors.go Resolved
Comment thread internal/cmds/cmds.go Resolved
Comment thread internal/cmds/cmds.go Resolved
Comment thread internal/cmds/cmds.go Resolved
Comment thread internal/cmds/cmds.go Resolved
Comment thread internal/cmds/cmds.go Resolved
Comment thread internal/cmds/cmds.go Resolved

@birukyitbarek birukyitbarek left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leverage copilot to do these reviews, if more context is needed happy to provide.
Also some of the comments were made 3 days ago but weren't published so if you made iteration changes since some of the comments might be out of line but should be easy to understand what code block they are referring to.

var goalStateKeys []types.GoalStateKey
for _, s := range goalStates {
for _, setting := range s.Settings {
if err := validateGoalStateSettings(setting); err != nil {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if one goal state has a missing ExtensionState, ExtensionName or SeqNo, validateGoalStateSettings rejects the whole batch. The ETag doesn't advance, so the service retries every 5 seconds forever and the valid goal states never run. A missing ExtensionState used to be allowed, so the check at :99 is now dead code. Skip or report only the bad entry.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment for line 217 below.

ctx.Log("error", fmt.Sprintf("error when trying to execute goal state: %v", e))
var executionErr *ImmediateCommandExecutionError
if errors.As(e, &executionErr) {
return executionErr.InstanceView.ExitCode, errors.Wrap(e, "error when trying to execute goal state")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this returns InstanceView.ExitCode, which is 0 when the script succeeded but reporting failed. The caller gets exit code 0 together with an error, so use a non-zero code there.

Comment thread internal/cmds/cmds.go Outdated
if err != nil && runErr == nil {
runErr, exitCode = err, constants.ExitCode_BlobCreateOrReplaceFailed
}
errorFilePosition, err = appendToBlob(stderrF, errorBlobSASRef, errorBlobAppendClient, errorFilePosition, ctx)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if appending output to the blob fails after a successful script, the run now fails with ExitCode_BlobCreateOrReplaceFailed. Customers will see successful scripts reported as failed.

Comment thread internal/exec/exec.go

scriptPath := cmd

commandArgs, err := SetEnvironmentVariables(cfg)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a Setenv failure now stops the script from running; before, the script still ran. That may be fine, but it's a behavior change worth calling out.

@@ -0,0 +1,316 @@
package errorutil

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

General comment on pattern:

Positive codes are user errors and negative codes are system errors. But that doesn't appear to be followed consistently.

Examples:
Similar OS failures get opposite classes:
• Deleting the data directory (:12) is a system error (-50), but creating it (:292) is a user error (5400).
• Opening stdout (:13) is a system error (-51), but closing it (:277) is a user error (5013).

Files the handler writes itself are classified as user errors:
• Os_InvalidTerminalGoalStatesJson (:288, 5208)
• Os_InvalidStoredSequenceNumber (:239, 4204)
• Os_InvalidProcessRecordFormat (:214, 3654)
Yet Internal_invalidMrseqFileSequenceNumber (:92, -5412) is a system error, so the same corrupt-state condition gets both classes.

Files written by the platform or guest agent are classified as customer errors:
• CustomerInput_missingCertificateThumbprint and CustomerInput_invalidProtectedSettingsBase64 (:168–169, used at handlersettingscommon.go:106)
• Os_InvalidHandlerEnvironmentJson (:163)
• CustomerInput_unexpectedRuntimeSettingsCount (handlersettingscommon.go:93)
Meanwhile, invalid JSON in the same file two lines up (handlersettingscommon.go:90) is Internal_badConfig, a system error

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I made this comment 3 days ago (but forgot to publish) and if you updated this file since, some of the line numbers might be out of whack.

statusItem, err := status.GetSingleStatusItem(ctx, statusType, c, msg, metadata.ExtName)
if statusType != types.StatusTransitioning {
var view types.RunCommandInstanceView
if err := json.Unmarshal([]byte(msg), &view); err != nil {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if json.Unmarshal fails on a finished command's result, the code returns before notifier.Notify. The final status is never sent to the host agent, though it always was before.

cfg, err := handlersettings.GetHandlerSettings(hEnv.HandlerEnvironment.ConfigFolder, extensionName, seqNum, ctx)
// Fail the extension status and handler invocation only when deployment failure is enabled and the command has a non-zero failure exit code.
cfg, err := readHandlerSettings(hEnv.HandlerEnvironment.ConfigFolder, extensionName, seqNum, ctx)
if err == nil && cfg.PublicSettings.TreatFailureAsDeploymentFailure && cmd.FailExitCode != 0 {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This block now returns an error, so main.go:33 exits with FailExitCode, which changes the handler's exit code. On main it effectively returned nil.
• A GetHandlerSettings failure is still silently ignored.
• The ClarificationError substatus has status "error" even when the top-level status is Success.

Comment thread internal/cmds/cmds.go
// Report the output streams to blobs
outputFilePosition, err = appendToBlob(stdoutF, outputBlobSASRef, outputBlobAppendClient, outputFilePosition, ctx)
errorFilePosition, err = appendToBlob(stderrF, errorBlobSASRef, errorBlobAppendClient, errorFilePosition, ctx)
outputFilePosition = appendOutput(stdoutF, outputBlobSASRef, outputBlobAppendClient, outputFilePosition)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if appending output to the blob fails after a successful script, the run now fails, so customers see successful scripts reported as failed.

Comment thread internal/exec/exec.go
exitCode = code
if signaled {
ctx.Log("message", "Timeout:"+err.Error())
return exitCode, errorutil.EnsureClarification(errorutil.CommandExecution_interruptedByVmShutdown, fmt.Errorf("command terminated with exit status=%d", exitCode))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: every process killed by a signal is reported as interruptedByVmShutdown. Timeouts, the out-of-memory killer and the user's own kill would all be misreported.

func ValidateScriptTypeAgainstPolicy(ctx *log.Context, scriptType handlersettings.ScriptType, allowedScriptTypesString string) error {
allowedScriptTypes, _ := StringToAllowedScriptTypeFlag(allowedScriptTypesString)
allowedScriptTypes, parseErr := StringToAllowedScriptTypeFlag(allowedScriptTypesString)
if parseErr != nil && allowedScriptTypesString != "" {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the message says the script type isn't allowed, but the real problem is a malformed policy file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants