Overview of incident
On Tuesday, May 24th an email was sent to the Active Directory email list (Subject : “New Config Manager NCSU-level OSD task sequences”) announcing the upcoming adjustment to the 20H2 Task Sequence changes, and detailed on the Active Directory website.
On Friday, May 27 the new OSD task sequences were deployed per the announcement on the Active Directory website.
When applying the new Windows 10 Education 20H2 Upgrade task sequence it was applied as required, as it had been previously, to the SUP-WSUS collections (which are used for monthly patches), that contain computers where most are already running 20H2 or newer. Windows 10 20H2 was set as the minimum supported Windows 10 version on July 9th, 2021 (CHG0030982). This Task Sequence will only apply to Windows 10 OS’s (so not Server OS or Windows 11).
Computers then did receive the new upgrade task sequence would have seen the task sequence progress dialog while it was running if someone was signed in.
There are five steps in the task sequence of which only three ran. The folder, “Upgrade the Operating System”, has two steps, “Upgrade Operating System” and “Restart Computer”, that only run if all three conditions are met:
- The Windows SKU does not contain “%LTS%”
- The build is less than 19042
- The OS is 64-bit and language is English
The second folder in the task sequence, “Upgrade Drivers”, however, did not have any conditions set such the computers that received the upgrade task sequence ran the Invoke-CMApplyDriverPackage.ps1 PowerShell script which is the script that checks for and installs drivers during OSD.
Windows 10 20H2 Upgrade is the name of the task sequence, was what displayed in the popup window, but is not indicative of what steps in the task sequence actually ran.ndows 10 20H2 Upgrade is the name of the task sequence and is not indicative of what steps in the task sequence actually ran.
Computers running a Windows build greater than or equal to 19042 did not perform any Windows upgrade since those steps were skipped. Any Windows 10 computer with an older OS build 19042 has been attempting a Windows OS upgrade multiple times per day since July 9th, 2021.
Any collections that have a defined maintenance window were likely to not have run the deployment since the deployment was done on a Friday morning. Systems that are mission critical should have maintenance windows defined.
Process changes to ensure that we don’t run into this issue:
The following changes are being made to address any issues uncovered during this incident:
- A new collection, NCSU-Devices-Windows 10-Required Feature Update, has been created whose membership query rule is any Windows build that is less than the currently required build — 19042 (20H2) as of this writing — and where the OS caption is “Microsoft Windows 10 Education”. In the future, required upgrade task sequences will be deployed to this collection and computers will drop out of the collection shortly after upgrading.
- Conditions will be set on the Upgrade Drivers folder in upgrade task sequences to only run if the computer is actually being upgraded. This step will also be pulled into a separate Task Sequence and nested into the OS Upgrade Task Sequences so that the code is in one and only one location.
- The email announcing the change was only sent to the Active Directory list which was an oversight on my part. In the future, emails will go to the Active Directory and NAG lists.
- Outage announcements for degraded performance, will be posted if we run into similar situations.