FIX: CMG stuck on Deleting? Check SQL permissions for the public server role!
Have you ever had the need to delete a Cloud Management Gateway (CMG) deployment from ConfigMgr?
If so, did it work as intended? ?
I’ve personally come across a few issues on the Azure side of things, where resources wouldn’t get deleted. Most times that was due to granular permissions on Resource Groups and was fixed by asking the appropriate administrator to remove the resources.
This time though, the Azure resources did get removed. But the status of CMG in the ConfigMgr Admin console showed “Deleting”
First thought was to give it some time. And then some more. A weekend has passed, and the CMG still seems stuck on deleting.
What’s going on?
Cleaning up (b)admin accounts in ConfigMgr
Recently, I overheard a conversation between IT staff members at one of my customers…
One colleague to another:
“… maybe we should reboot those servers first, because there might still be processes running as the badmin account that we renamed.”
Me, with sparked interest since I heard the badmin account referenced:
“Oh, good to know you renamed that account. You do know that account is used in your SCCM environment, don’t you?”
Customer, thinking it, but not saying it:
“I didn’t think of that. But I haven’t noticed all hell breaking loose on us so we’re good. I’ll just nod yes in a reassuring way…”
A use case for CMPivot: look for installed software FAST!
This is an extensive explanation on how CMPivot can help you pull information from clients in Configuration Manager. Specifically, we’ll lookup an MSI Product Code for use with the Detection Method for an Application Deployment Type. But it covers some technical background and hints I hope you’ll find useful.
TL;DR If you already know about CMPivot and remote PowerShell and WMI, skip ahead to the CMPivot queries. If not, read on!
What is CMPivot?
If you’re on Configuration Manager CB 1806 or later, you have access to the CMPivot feature. If you’ve never used it, just select a Device Collection in the CM Console and click Start CMPivot in the Actions Ribbon. You can’t go wrong playing around with it as it only queries devices for stuff in a read-only fashion. But maybe start with a limited collection to avoid high resource impact.
When you open the CMPivot tool, the Welcome message has a good description of what it can be used for:
Note that the possibilities of CMPivot in CM 1806 were rather limited, as it was the first release of the feature. Still, it was good enough to be very efficient in countering a malware attack. And since 1810, we can query hardware inventory instances, including the custom one you defined in Client Settings.
See the full documentation on CMPivot here.
Live life in the fast lane
Because CMPivot queries a device in real-time, it is much faster than Hardware Inventory, which by default only pulls data from your clients once every 7 days. Even if you change that to once daily, it’s still too slow to help you detect change in the system’s software repository after a software deployment. Also, Hardware Inventory will pull everything you tell it to, as configured in the Client Settings. All that data easy accumulates to multiple MegaBytes per device. Multiplied by the number of client devices, this has an impact on bandwidth and resource usage on your Site Server and Site database.
CMPivot will only query the portion of data you’re after, on the limited number of devices you select, and it won’t store it in the database. And when you query on hardware inventory, it will pull the latest data from the Site database and update it based on CMPivot query results. This makes CMPivot instant, lightweight and powerful!
Once you’ve got the results you’re after, you can use them as a dataset and export to CSV or copy to the clipboard for further manipulation. Or, you can create a collection from the resulting set of devices and perform whatever action is needed to rectify a situation or further monitor it.
But the best thing is, you can target a PowerShell script to those systems, directly from the CMPivot screen. Scripts are pushed and followed up on through the fast channel as well, so you can perform instant action on your instant results. You can only select the scripts you’ve already created and had approved in your CM site though. Would be cool to enter PS code and run it right from CMPivot… Which is a very bad idea, which we WON’T put up on uservoice!
Seriously. Let’s not do that. If you’re not sure why that’s a bad idea, contact us for an offline bashing discussion.
Application Detection Method
What follows is how can we put CMPivot to good use in the context of Application Deployment and specifically, Detection Methods.
I prefer to detect the presence of an application through the MSI product code and version property. This of course depends on the type of application. If you have the MSI in the source content, you can use the Detection Rule wizard to read its properties and -shazam- you’re done. However, if you have a setup.exe or similar file and you can’t extract its contents to retrieve a .msi file from it, things are a little more complex. Sure, you can use file system or registry detection. Or use PowerShell – it can do anything! But I found a cool way to detect the MSI properties quick and easy through CMPivot.
First, I use a Windows machine to deploy and test packages and applications on. This can be a virtual machine which I can easily revert to a clean state, or a physical machine. Doesn’t matter in this context, as long as it has a CM client. Minimum client version must be 1806 to make use of CMPivot. It needs to be on 1810 or later to query hardware inventory!
Second, we need an application… I’ll be deploying the Sophos Endpoint. It uses the type of installer as described above and when I inspect its contents through 7-Zip, I see nothing but meaningless files. At least, to me they’re meaningless.
I always check the available switches or parameters of an installer. Sometimes there’s an administrative install method or an extract option. Not in this case. And no other switch that might help me to detect the application.
In cases like these, I either look at a machine that already has the application, or I install it on my packaging machine. I then look at either the Windows registry or the Add/Remove Programs inventory to get the MSI code, if any. That’s a manual chore. Which I despise. So I revert to PowerShell and I perform a remote query on the client. Doing this from my workstation, allows me to easily copy/paste the retrieved information into the CM Console.
The WMI class that is often queried for this purpose, is Win32_Product.
Bad practice alert!
A query to the Win32_Product class triggers Windows Installer to check for inconsistencies and triggers repairs. This is described in the MSDN documentation on Win32_Product and is mentioned all over the interwebz.
Now, since we have the CM Client installed, we have a much better WMI Class (or CIM instance) to query for installed software:
It’s quicker to query and it doesn’t trigger any of the naughty side-effects of Win32_Product. So now we can use PowerShell or WMI Explorer or your weapon of choice to query for software remotely. This does, however, require WinRM to be configured to accept connections. Firewalls may get in the way as well.
CMPivot just works
It uses the fast communication channel to trigger the CM Client from the Site. It then reports back through state messages or, if you’re on CM 1810 and output is less than 80 KB, back through the fast channel.
So back to the case at hand. I’m at a customer with a rather strict security implementation, which does not allow for remote WMI queries or remote PowerShell instructions. I can setup a remote PSSession however, and here’s how I query for any Sophos products installed:
For reference, this query took 436 milliseconds. Fast enough for my standards. But it did take me a few minutes to get the output I was looking for. And I like to think I’m experienced on the matter.
All the same, I can now copy/paste the MSI code for the Sophos Endpoint from the SoftwareCode column, into the Detection Rule wizard. I also like to put a rule on the Product Version to be greater than or equal to a specific version, so I also copy/paste the ProductVersion value. And this is what my Application Detection Rule now looks like:
Splendid, that’ll do the trick!
So for remote PowerShell or WMI queries to work, there’s a few dependencies. And I need to know how to format output in PowerShell to get a usable result.
Now with CMPivot, our job got a lot easier. All you need is the CM client and at least PowerShell 4 on the target systems.
Again, the docs have all the details if you’re ready to learn more.
Here’s how easy it is.
Note: in my example, only 1 device reports back. If you select a collection with many online devices, you will get back as many results. Take this into account for resource usage. On the other hand, getting more results than what you strictly need, might actually help you to find the right information as you can compare results from multiple devices.
- Select a Device Collection where the test client is a member
- Start CMPivot from the Actions Ribbon or the right-click context menu
- Right-click the InstalledSoftware entity and click Insert
Here are two ways to filter and get the desired result. (video edited to trim idle time between query execution and results showing up)
1. Query all software and right-click to filter on Publisher
2. Type query to get results directly
- Copy/paste the SoftwareCode and ProductVersion values into the Detection Rule wizard
Hints when working with CMPivot queries
- Refer to the Welcome page and the product documentation for syntax and limitations
- The query pane uses IntelliSense: type a letter, use arrows to select and hit tab
- The wildcard character is ‘%’ not ‘*’
- Use the ‘back’ and ‘forward’ arrows to move through your query history
- Queries will timeout after an hour for devices that don’t report back
That’s it. No more waiting for Hardware Inventory to be reported back. No firewalls getting in the way. Speed and power!
If only election results and tax refunds came back this fast…
I had fun creating this blog post and I hope you found it useful!
Microsoft Defender ATP & Configmgr CMPivot with a CMG (Cloud Management Gateway) better together; How we saved the customer from Emotet related malware
In September 2018, one of our customers was targeted by a phishing attack in attempts to infiltrate malicious code on their systems. These attempts were successful and a widespread Emotet-related malware attack followed. This is the story on how we countered and contained it with the use of Configuration Manager CB 1806 and CMPivot. One of the tricky parts was that most workstations where still Windows 7 and Windows 7 embedded. Only 20% was native Windows 10 1709.
The phishing email was well disguised and was made to believe it originated from a corporate director. The content of the email suggested that invoices had not been paid and requested the addressee to open attached Excel or PDF file and follow up. This was convincing enough to some, who indeed opened the files and clicked on whatever links or “Enable content” messages held within. This triggered the seemingly random creation of executables on the local system, which in turn contacted Command & Control servers to download further payloads.
First signs of infection came on a Tuesday, through SCCM Endpoint Protection alerts: 5 infections of an Emotet-related strain, which were all quarantined. This was reported to the Security Team lead, with no further actions other than following up on more infection alerts, if any. On Wednesday, there were about 25 reported infections and we were actively investigating.
We used the (at the time) new CMPivot feature in SCCM CB 1806 to detect malware-generated executable files which served to contact C&C and download payloads. Through SCCM Endpoint Protection, we learned there was a pattern in the filenames and the locations where they were created. Some of these executables were used to create services which ran under SYSTEM context. We then triggered PowerShell scripts on the detected systems, to stop and disable those services and to move malicious files to a holding locations. This approach rendered the files harmless, while still allowing to collect and investigate them further. We also wanted to report on systems where users had non-authorized local admin rights.
These are the queries we used in CMPivot:
Process | where ((CommandLine like ‘%system32//%.exe’) or (CommandLine like ‘%syswow64//%.exe’))
Process | where (CommandLine == ‘”C:\\Windows\\system32\\volumebundle.exe”‘)
Process | where (CommandLine == ‘”C:\\Windows\\system32\\volumebundle.exe”‘) | where (CreationDate == ‘2018-09-14 06:41:00’)
Process | where (CreationDate like ‘2018-09-14 06:41:00’)
Administrators | where (ObjectClass == ‘User’) | where (not (Name like ‘%Admin%’))
If wanted we can deliver the PS script to mitigate infections.
That same day, the Network & Security team reported a lot of C&C related activity being picked up by the CheckPoint firewall. In the days following, the outgoing traffic was monitored and firewalls kept most, if not all suspicious traffic contained. Malware infections kept spreading however and the phishing attack was still going as well. Teams were informed and staff was instructed to not open any suspected emails.
First reports came is that users had received alerts from cloud services that their credentials had been compromised. For instance, Google accounts had been used to login from IP addresses throughout the world. These credentials had been stored on the users’ computers in browser password managers.
That Wednesday evening, a crisis meeting with corporate management was held and reactive measures were discussed. The only real actions we felt we could suggest, was to cut off internet access for the time being and to shut down SMB v1 services. The first would impact business-customer productivity and was denied. The latter would severely impact business productivity as most file servers and legacy applications still used SMB v1. This suggestion was also denied. For reference: there had been a WannaCry-related attack the year before, where SMB v1 was cut off by the IT teams to prevent the attack from spreading further, which was promptly overruled by business the next day, accepting the risk of the malware spreading, in fact waving away the risk as overrated. Luckily, no files or systems were encrypted then.
On Thursday, we proposed to implement Windows Defender ATP, however, another service provider implemented Sentinel One implementation. It did not go well as prerequisites were not advertised installed correctly. Windows Embedded was not mentioned as not being supported, AFTER the wide spread deployment of SentinelOne agents through SCCM application deployment. Furthermore, despite SentinelOne actively detecting and stopping many malware strains, infections were still spreading and firewall kept picking up C&C traffic.
In the meantime, the proposed WDATP PoC was setup and agents were onboarded. However, it clashed with the still present SentinelOne agents, reporting it as malware and rendering slower systems unusable due to the high usage of CPU resources. We finally found a method to uninstall SentinelOne agents. Again, we used CMPivot to detect and intervene on systems where this issue occurred. The results were used to create a device collection to deploy the uninstall method to.
These are the queries we used in CMPivot:
EventLog(‘System’) | where (EventID == 7031) and (Message like ‘%Sentinel%’)
EventLog(‘System’) | summarize countif( (Message like ‘The Sentinel Agent service terminated unexpectedly.%’) ) by Device | where (countif_ > 0)
The cleanup of SentinelOne was mostly successful through these methods, leaving some systems to be reinstalled completely.
ATP onboarding was successful and immediately, new malware infections were reported and contained, including strains the other product had not detected.
In the end, there were no confirmed reports of data breaches other than some users personal credentials having been stolen from browser password managers.
WDATP was purchased after a positive evaluation and thanks to Microsoft’s technical and commercial approach in supporting the deployment and CMPivot was a great addition to react when we where out of control.
If you want to look at Microsoft Defender Advanced Threat Protection , please do not hesitate to contact us at firstname.lastname@example.org
Best Regards ,
The workplace & security team.
Hybrid Mobile Device Management (MDM) and why you should plan for migration now !
Microsoft recently posted a service change announcement in the Office Message Center to customers using Hybrid Mobile Device Management (MDM). Maybe you are one of those type of customers that need to take action as you are using SCCM + Intune in hybrid mode and this impacts you .
MC146431 – Plan for Change: Move to Intune on Azure for your Mobile Device Management
Since launching on Azure over a year ago, Intune has added hundreds of new customer-requested and market-leading service capabilities, and now offers far more capabilities than those offered through hybrid Mobile Device Management (MDM). Intune on Azure provides a more integrated, streamlined administrative experience for your enterprise mobility needs.
As a result, we see that most Enterprise Mobility + Security (EMS) customers choose Intune on Azure over hybrid MDM. The number of customers using hybrid MDM continues to decrease as more customers move to the cloud. Therefore, on September 1, 2019, we will retire the hybrid MDM service offering.
Note: This change does not affect on-premises System Center Configuration Manager (ConfigMgr) or co-management for Windows 10 devices.
Please plan your migration to Intune on Azure for your MDM needs. OB-V-US has tools and other resources to help you with this migration.
Best Regards ,
Upgrading from ConfigMgr 1702 to 1706 gives you “Microsoft SQL Server reported SQL message 2627, severity 14” in your status messages
If you upgraded your ConfigMgr 1702 or earlier environment to Configmgr 1706 and in the status messages after the upgrade you get :
Microsoft SQL Server reported SQL message 2627, severity 14: [Microsoft][SQL Server Native Client 11.0][SQL Server]Violation of PRIMARY KEY constraint ‘SUM_DriverUpdates_PK’. Cannot insert duplicate key in object ‘dbo.SUM_DriverUpdates’. The duplicate key value is (d8483f4f-0390-49db-b251-faf884dd8eaf
Be aware that the Product Group are aware of this issue and are on a working on a fix. The result of this problem is that admins cannot see new Surface driver updates.
Other than that, nothing else is impacted so it can be ignored.
Hope it helps ,
MVP Enterprise Mobility
Enterprise Mobility : In the land of NDES – Where one eye is King and you need to watch your CRL Delta files
I was doing a EMS POC and deployment of certificates on mobile devices was a requirement. So I needed to setup an NDES server with a separate Subordinate CA for MDM , NDES Server and SCCM Certificate Registration Point (CRP). Big deal I thought as I did it a already multiple times. At my customer we worked close with the server team and setup the infrastructure which was working fine at first sight.
After a reboot of the NDES server I was struggling to get the Network Device Enrollment Service (NDES) up and running again as it would throw me an error 500.
The event log of the NDES Server told me the following:
The Network Device Enrollment Service cannot retrieve one of its required certificates (0x80070057). The parameter is incorrect.
The Network Device Enrollment Service cannot be started (0x80070057). The parameter is incorrect.
When the service starts, it searches for two certificates that are used by the service :
1. The service searches in the machine MY store AND
2. The certificate must have the following extensions AND
For the Key Exchange certificate:
– ExtendedKeyUsage: “Certificate Request Agent”
– KeyUsage: Encryption (0x20)
For the enrollment agent certificate:
– ExtendedKeyUsage: “Certificate Request Agent”
– KeyUsage: Signature (0x80)
3. The certificate must not be archived AND
4. The computer must have the private key for the certificate AND
5. The certificate must be issued by the same CA that the service is configured for AND
6. The certificate must have a valid chain AND
7. If there is more than one certificate for either of the certificates that meets the previous criteria, the service will select the most recent one (the latest that was issued)
Troubleshooting certificate issues will require you to enable the CryptoAPI 2.0 Event Logging :
The CryptoAPI 2.0 Diagnostics is a feature available from Windows Server 2008 that supports the trouble shooting of issues concerned with:
– Certificate Chain Validation
– Certificate Store Operations
– Signature Verification
Enable CAPI2 logging by opening the Event Viewer and navigating to the Event Viewer (Local)\Applications and Services Logs\Microsoft\Windows\CAPI2 directory and expand it. You should see a view named Operational.Next, right-click on the Operational view and click the Enable Log menu item.
Searching the right information under the capi2 operational log :
I was quite sure, that I was able to download the CRL (Certificate Revocation List) and I double checked that by browsing to the URL ‘.crl”>http://pki.xxx.be/CertEnroll/<NameOfYourSubCA>.crl’ and I was able to download the file. When digging deeper in the eventID’s , I found at EventID 42 , the following URL ‘.crl”>http://pki.xxx.be/CertEnroll/<NameOfYourSubCA+>.crl’ was shown. This means it was looking for the availability of the Delta CRL, which was visible on the web site of my CRL:
When I finally tried to download this CRL Delta file, it failed. I remembered myself, that IIS is treating the + sign very differently in URL’s.
I needed to set the setting “Allow double escaping” in the web.config file as shown below :
After enabling this, NDES was able to retrieve the Delta CRL file and start the service gracefully.
So if your NDES Server is throwing “The Network Device Enrollment Service cannot retrieve one of its required certificates (0x80070057). The parameter is incorrect.”, do not only check your certificates on the Server, check also your CRLs and Delta CRLs!
Hope it Helps,
MVP Enterprise Mobility