Blog

Tenant Attach, Applications and how it can bring value for your helpdesk

Recently we (Kenny & myself) engaged with a bigger customer to migrate multiple MEMCM environments into a new standard platform, the goal of this project is to unify the multiple companies.

One of our challenges was the Application delivery strategy. It was a heavy discussed topic as we faced the typical topic of “Application – Collection – AD Group methodology”, why change something that has worked for so many years ?

ServiceNow today is providing an application catalog for users to request their software, but once the software has been requested a ticket gets created and delivered to the helpdesk queue ==> where an engineer takes the user or device object and populates it in an AD Group. I guess most of us are familiar with this, since it’s probably the most common setup seen in a lot of companies.

As we are getting more and more user driven improvements in the latest version(s) of ConfigMgr, we wanted to take advantage of the nice and shiny new application approval process. Like the product team promotes, we want to avoid creating 500+ collections and instead be like cool kids going almost collection-less!

The goal was set, to go almost collection-less and have it fully automated using ServiceNow as the “orchestration” tool which will approve / trigger the approval requests. Due to hectic Project Planning etc. the full automation part will be integrated later this year, however we needed something now so that we could keep our “vision” of going collection-less and avoid creating any more application AD Groups!

This is where we thought Tenant Attach would come into play, we are not going to explain what Tenant Attach is. There are many good blogs about it.

One thing of Tenant Attach we are focused on is the “Application (Preview)” feature. Could this provide us with a temporary solution to approve / push the application installation part – to avoid setting up all these AD Groups and Collections again?

At our customer’s helpdesk, they don’t have real administrative privileges in the CM console. They are just capable of doing the normal helpdesk stuff (remote control, reporting + HW Inventory).

Now we have explained our scenario, let’s go technical.

We have our Helpdesk Group, which in our lab environment has 1 user in it called “helpdesk”

This user is synced as a hybrid identity to AAD.
Shoutout to Microsoft: it would be cool that there would be a possibility to use AAD Only accounts in CM console .. a lot of customers have on-prem only Admin accounts & AAD Only Admin accounts!

I’ve setup a few test deployments to my “test – user collection”, 2 of them require approval (based on client settings they will be hidden in software center)

Our test device:

So, that’s looking good. Let’s go into Tenant Attach now (everything has already been setup & synced)

We’ll be using the helpdesk account, since he/she will be “handling the ticket”:

Let’s open the application (preview) feature

The user targeted applications are visible, you already notice that we have more applications visible compared to what the actual user would be able to see in software center (this because we hide the apps that would need approval)

Good .. now let’s try to install 7-zip

Hmmm, not what we were hoping for .. time to open the logs. Browsing the documentation it seems that we should look the adminservice.log

Ok, so the adminservice is trying to execute the SMS_ApplicationRequest.CreateApprovedRequest as my Helpdesk user .. so this means that my helpdesk user does not have enough privileges in ConfigMgr to execute this task.

Now the tricky Part, and this will be the case in most companies, what if your helpdesk doesn’t have any real administrative privileges in Configuration Manager ? as you already noticed our helpdesk user only has “Read Only Analyst + Remote Tools Operator”, we don’t really want to give them more.

Back to the documentation part, this brings us here

So let’s create a custom security role for this and link it to our helpdesk group:

Back to Tenant Attach & let’s try again.

Oh yeah, this is looking great.

Ok cool, but now let’s try it for one of the hidden apps. Because that’s the main goal behind this exercise.

Yes! It works

Checking the console now, we can see that following approvals have been created & approved via the Adminservice.

If you’re still reading beyond this point, that’s it. A (not so) quick write-down of what we have been trying to accomplish.

This is a really cool feature, that i’m convinced about will bring a lot of value to your helpdesk. You can enable your helpdesk to do these actions without having a footprint on the CM console. (using your phone for example when your out for grocery shopping)

If you’re a small shop, and you’re not looking to invest in a ServiceNow or other tool, this is an easy and transparent way to empower your helpdesk.
Bigger shops who’ve been looking for a way to get away from those pesky install/uninstall collections. look into it, you might have the same use-case as we have just worked out.

Kim, out.

My way of keeping track of changes in Task Sequences

Along my years as a consultant. I have made several attempts at tracking changes in ConfigMgr Task Sequences. The obvious one being a note block and pen. Then, OneNote came along… Yes, I was taking notes before OneNote was born, Yes I’m aging. Thank you for that thought.

In the end, there is one method that stuck with me as useful and consistent throughout versions of Configuration Manager. I’ve had my colleagues and customers asking me about it and adopting it in their ways of maintaining Task Sequences. So I wanted to share it with you as well.

(more…)

FIX: CMG stuck on Deleting? Check SQL permissions for the public server role!

Have you ever had the need to delete a Cloud Management Gateway (CMG) deployment from ConfigMgr?

If so, did it work as intended? ?

I’ve personally come across a few issues on the Azure side of things, where resources wouldn’t get deleted. Most times that was due to granular permissions on Resource Groups and was fixed by asking the appropriate administrator to remove the resources.

This time though, the Azure resources did get removed. But the status of CMG in the ConfigMgr Admin console showed “Deleting”

First thought was to give it some time. And then some more. A weekend has passed, and the CMG still seems stuck on deleting.

What’s going on?

(more…)

Cleaning up (b)admin accounts in ConfigMgr

Recently, I overheard a conversation between IT staff members at one of my customers…

One colleague to another:

“… maybe we should reboot those servers first, because there might still be processes running as the badmin account that we renamed.”

Me, with sparked interest since I heard the badmin account referenced:

“Oh, good to know you renamed that account. You do know that account is used in your SCCM environment, don’t you?”

Customer, thinking it, but not saying it:

“I didn’t think of that. But I haven’t noticed all hell breaking loose on us so we’re good. I’ll just nod yes in a reassuring way…”

(more…)

A use case for CMPivot: look for installed software FAST!

This is an extensive explanation on how CMPivot can help you pull information from clients in Configuration Manager. Specifically, we’ll lookup an MSI Product Code for use with the Detection Method for an Application Deployment Type. But it covers some technical background and hints I hope you’ll find useful.

TL;DR If you already know about CMPivot and remote PowerShell and WMI, skip ahead to the CMPivot queries. If not, read on!

What is CMPivot?

If you’re on Configuration Manager CB 1806 or later, you have access to the CMPivot feature. If you’ve never used it, just select a Device Collection in the CM Console and click Start CMPivot in the Actions Ribbon. You can’t go wrong playing around with it as it only queries devices for stuff in a read-only fashion. But maybe start with a limited collection to avoid high resource impact.

When you open the CMPivot tool, the Welcome message has a good description of what it can be used for:

Note that the possibilities of CMPivot in CM 1806 were rather limited, as it was the first release of the feature. Still, it was good enough to be very efficient in countering a malware attack. And since 1810, we can query hardware inventory instances, including the custom one you defined in Client Settings.

See the full documentation on CMPivot here.

Live life in the fast lane

Because CMPivot queries a device in real-time, it is much faster than Hardware Inventory, which by default only pulls data from your clients once every 7 days. Even if you change that to once daily, it’s still too slow to help you detect change in the system’s software repository after a software deployment. Also, Hardware Inventory will pull everything you tell it to, as configured in the Client Settings. All that data easy accumulates to multiple MegaBytes per device. Multiplied by the number of client devices, this has an impact on bandwidth and resource usage on your Site Server and Site database.

CMPivot will only query the portion of data you’re after, on the limited number of devices you select, and it won’t store it in the database. And when you query on hardware inventory, it will pull the latest data from the Site database and update it based on CMPivot query results. This makes CMPivot instant, lightweight and powerful!

Once you’ve got the results you’re after, you can use them as a dataset and export to CSV or copy to the clipboard for further manipulation. Or, you can create a collection from the resulting set of devices and perform whatever action is needed to rectify a situation or further monitor it.

But the best thing is, you can target a PowerShell script to those systems, directly from the CMPivot screen. Scripts are pushed and followed up on through the fast channel as well, so you can perform instant action on your instant results. You can only select the scripts you’ve already created and had approved in your CM site though. Would be cool to enter PS code and run it right from CMPivot… Which is a very bad idea, which we WON’T put up on uservoice!

Seriously. Let’s not do that. If you’re not sure why that’s a bad idea, contact us for an offline bashing discussion.

Application Detection Method

What follows is how can we put CMPivot to good use in the context of Application Deployment and specifically, Detection Methods.

I prefer to detect the presence of an application through the MSI product code and version property. This of course depends on the type of application. If you have the MSI in the source content, you can use the Detection Rule wizard to read its properties and -shazam- you’re done. However, if you have a setup.exe or similar file and you can’t extract its contents to retrieve a .msi file from it, things are a little more complex. Sure, you can use file system or registry detection. Or use PowerShell – it can do anything! But I found a cool way to detect the MSI properties quick and easy through CMPivot.

First, I use a Windows machine to deploy and test packages and applications on. This can be a virtual machine which I can easily revert to a clean state, or a physical machine. Doesn’t matter in this context, as long as it has a CM client. Minimum client version must be 1806 to make use of CMPivot. It needs to be on 1810 or later to query hardware inventory!

Second, we need an application… I’ll be deploying the Sophos Endpoint. It uses the type of installer as described above and when I inspect its contents through 7-Zip, I see nothing but meaningless files. At least, to me they’re meaningless.

I always check the available switches or parameters of an installer. Sometimes there’s an administrative install method or an extract option. Not in this case. And no other switch that might help me to detect the application.

In cases like these, I either look at a machine that already has the application, or I install it on my packaging machine. I then look at either the Windows registry or the Add/Remove Programs inventory to get the MSI code, if any. That’s a manual chore. Which I despise. So I revert to PowerShell and I perform a remote query on the client. Doing this from my workstation, allows me to easily copy/paste the retrieved information into the CM Console.

The WMI class that is often queried for this purpose, is Win32_Product.

Bad practice alert!

A query to the Win32_Product class triggers Windows Installer to check for inconsistencies and triggers repairs. This is described in the MSDN documentation on Win32_Product and is mentioned all over the interwebz.

Now, since we have the CM Client installed, we have a much better WMI Class (or CIM instance) to query for installed software:
root\cimv2\sms\SMS_InstalledSoftware

It’s quicker to query and it doesn’t trigger any of the naughty side-effects of Win32_Product. So now we can use PowerShell or WMI Explorer or your weapon of choice to query for software remotely. This does, however, require WinRM to be configured to accept connections. Firewalls may get in the way as well.

CMPivot just works

It uses the fast communication channel to trigger the CM Client from the Site. It then reports back through state messages or, if you’re on CM 1810 and output is less than 80 KB, back through the fast channel.

So back to the case at hand. I’m at a customer with a rather strict security implementation, which does not allow for remote WMI queries or remote PowerShell instructions. I can setup a remote PSSession however, and here’s how I query for any Sophos products installed:

For reference, this query took 436 milliseconds. Fast enough for my standards. But it did take me a few minutes to get the output I was looking for. And I like to think I’m experienced on the matter.

All the same, I can now copy/paste the MSI code for the Sophos Endpoint from the SoftwareCode column, into the Detection Rule wizard. I also like to put a rule on the Product Version to be greater than or equal to a specific version, so I also copy/paste the ProductVersion value. And this is what my Application Detection Rule now looks like:

Splendid, that’ll do the trick!

So for remote PowerShell or WMI queries to work, there’s a few dependencies. And I need to know how to format output in PowerShell to get a usable result.

Now with CMPivot, our job got a lot easier. All you need is the CM client and at least PowerShell 4 on the target systems.
Again, the docs have all the details if you’re ready to learn more.

CMPivot queries

Here’s how easy it is.

Note: in my example, only 1 device reports back. If you select a collection with many online devices, you will get back as many results. Take this into account for resource usage. On the other hand, getting more results than what you strictly need, might actually help you to find the right information as you can compare results from multiple devices.

Steps

  • Select a Device Collection where the test client is a member
  • Start CMPivot from the Actions Ribbon or the right-click context menu
  • Right-click the InstalledSoftware entity and click Insert
Here are two ways to filter and get the desired result. (video edited to trim idle time between query execution and results showing up)

1. Query all software and right-click to filter on Publisher

2. Type query to get results directly

  • Copy/paste the SoftwareCode and ProductVersion values into the Detection Rule wizard

Hints when working with CMPivot queries

  • Refer to the Welcome page and the product documentation for syntax and limitations
  • The query pane uses IntelliSense: type a letter, use arrows to select and hit tab
  • The wildcard character is ‘%’ not ‘*’
  • Use the ‘back’ and ‘forward’ arrows to move through your query history
  • Queries will timeout after an hour for devices that don’t report back

That’s it. No more waiting for Hardware Inventory to be reported back. No firewalls getting in the way. Speed and power!

If only election results and tax refunds came back this fast…

I had fun creating this blog post and I hope you found it useful!

Microsoft Defender ATP & Configmgr CMPivot with a CMG (Cloud Management Gateway) better together; How we saved the customer from Emotet related malware

In September 2018, one of our customers was targeted by a phishing attack in attempts to infiltrate malicious code on their systems. These attempts were successful and a widespread Emotet-related malware attack followed. This is the story on how we countered and contained it with the use of Configuration Manager CB 1806 and CMPivot. One of the tricky parts was that most workstations where still Windows 7 and Windows 7 embedded. Only 20% was native Windows 10 1709.

The phishing email was well disguised and was made to believe it originated from a corporate director. The content of the email suggested that invoices had not been paid and requested the addressee to open attached Excel or PDF file and follow up. This was convincing enough to some, who indeed opened the files and clicked on whatever links or “Enable content” messages held within. This triggered the seemingly random creation of executables on the local system, which in turn contacted Command & Control servers to download further payloads.

First signs of infection came on a Tuesday, through SCCM Endpoint Protection alerts: 5 infections of an Emotet-related strain, which were all quarantined. This was reported to the Security Team lead, with no further actions other than following up on more infection alerts, if any. On Wednesday, there were about 25 reported infections and we were actively investigating.

We used the (at the time) new CMPivot feature in SCCM CB 1806 to detect malware-generated executable files which served to contact C&C and download payloads. Through SCCM Endpoint Protection, we learned there was a pattern in the filenames and the locations where they were created. Some of these executables were used to create services which ran under SYSTEM context. We then triggered PowerShell scripts on the detected systems, to stop and disable those services and to move malicious files to a holding locations. This approach rendered the files harmless, while still allowing to collect and investigate them further. We also wanted to report on systems where users had non-authorized local admin rights.

 

These are the queries we used in CMPivot:

Process | where ((CommandLine like ‘%system32//%.exe’) or (CommandLine like ‘%syswow64//%.exe’))

Process | where (CommandLine == ‘”C:\\Windows\\system32\\volumebundle.exe”‘)

Process | where (CommandLine == ‘”C:\\Windows\\system32\\volumebundle.exe”‘) | where (CreationDate == ‘2018-09-14 06:41:00’)

Process | where (CreationDate like ‘2018-09-14 06:41:00’)

File(‘%public%\\*.exe’)

Administrators | where (ObjectClass == ‘User’) | where (not (Name like ‘%Admin%’))

 

If wanted we can deliver the PS script to mitigate infections.

That same day, the Network & Security team reported a lot of C&C related activity being picked up by the CheckPoint firewall. In the days following, the outgoing traffic was monitored and firewalls kept most, if not all suspicious traffic contained. Malware infections kept spreading however and the phishing attack was still going as well. Teams were informed and staff was instructed to not open any suspected emails.

First reports came is that users had received alerts from cloud services that their credentials had been compromised. For instance, Google accounts had been used to login from IP addresses throughout the world. These credentials had been stored on the users’ computers in browser password managers.

That Wednesday evening, a crisis meeting with corporate management was held and reactive measures were discussed. The only real actions we felt we could suggest, was to cut off internet access for the time being and to shut down SMB v1 services. The first would impact business-customer productivity and was denied. The latter would severely impact business productivity as most file servers and legacy applications still used SMB v1. This suggestion was also denied. For reference: there had been a WannaCry-related attack the year before, where SMB v1 was cut off by the IT teams to prevent the attack from spreading further, which was promptly overruled by business the next day, accepting the risk of the malware spreading, in fact waving away the risk as overrated. Luckily, no files or systems were encrypted then.

On Thursday, we proposed to implement Windows Defender ATP, however, another service provider implemented Sentinel One implementation. It did not go well as prerequisites were not advertised installed correctly. Windows Embedded was not mentioned as not being supported, AFTER the wide spread deployment of SentinelOne agents through SCCM application deployment. Furthermore, despite SentinelOne actively detecting and stopping many malware strains, infections were still spreading and firewall kept picking up C&C traffic.

In the meantime, the proposed WDATP PoC was setup and agents were onboarded. However, it clashed with the still present SentinelOne agents, reporting it as malware and rendering slower systems unusable due to the high usage of CPU resources. We finally found a method to uninstall SentinelOne agents.  Again, we used CMPivot to detect and intervene on systems where this issue occurred. The results were used to create a device collection to deploy the uninstall method to.

 

These are the queries we used in CMPivot:

EventLog(‘System’) | where (EventID == 7031) and (Message like ‘%Sentinel%’)

EventLog(‘System’) | summarize countif( (Message like ‘The Sentinel Agent service terminated unexpectedly.%’) ) by Device | where (countif_ > 0)

 

The cleanup of SentinelOne was mostly successful through these methods, leaving some systems to be reinstalled completely.

ATP onboarding was successful and immediately, new malware infections were reported and contained, including strains the other product had not detected.

In the end, there were no confirmed reports of data breaches other than some users personal credentials having been stolen from browser password managers.

WDATP was purchased after a positive evaluation and thanks to Microsoft’s technical and commercial approach in supporting the deployment and CMPivot was a great addition to react when we where out of control.

If you want to look at Microsoft Defender Advanced Threat Protection , please do not hesitate to contact us at info@ob-v-us.be

Best Regards ,

The workplace & security team.

Attending MVP summit 2019

For those who don’t know the MVP summit. This is the annual gathering of all MVP’s at the Microsoft campus in Redmond.

For those who KNOW the MVP summit they know that you are under strict NDA not to tell anyone outside of the MVP program where we are heading with different aspects of Microsoft products.

What I can show though how awesome it is to connect with so many people around the globe and meet them here at Microsoft to discuss different topics regarding the technologies we work with every day.

image

How to query custom logs data in Log analytics

This post is a follow-up on how to SCCM custom data into your log analytics environment.

As soon as you have your SCCM custom logs, or any other logs, in log analytics they get indexed under the type you have specified.

In this particular case I used SCCMLOG_CL (note that the CL is mandatory). So lets jump into the log analytics query window to find out what’s in the logs at this time:

Browse to Log analytics => Logs

clip_image002

The log analytics query window will open and will give you the opportunity to start your query journey:

clip_image004

Remember our custom type: SCCMLOGS_CL. Note the autosuggest feature which will help you to create your own queries

clip_image006

If you run this query you will get all the results within the type. This is a great way to check whether data is flying in.

clip_image008

So now we’ll start finding more in detail patterns. If you type where in the next line you’ll get all the fields in your data:

clip_image010

Let’s select Rawdata where the word “error” is in the line:

clip_image011

So we get already a lot of results:

clip_image013

So another trick in your sleeve. You don’t need to type everything. It’s a point and click world to combine your query. Just click the + sign next to a field. In this case “Computer”.

clip_image015

This will add the field AND the content of the field to your query:

clip_image017

So now you can really start building your searches on your custom data.

Next time we’ll go over how you can actually create custom fields you can search on.

How to upload SCCM logs in Log Analytics

One of the great powers and conveniences of having all logs in 1 place is in fact that they are getting indexed and you can query them for different scenarios.

Just recently I was working on a project together with SCCM engineers and they basically told me a couple of times “it’s in this or that logfile”, they fire up SCCMtrace and start looking for the specific entry and start troubleshooting from there.

“OK” I thought, maybe just maybe there’s a better solution. Because of my monitoring background I don’t like to think reactive as in “it already happened” but love to think proactive.

clip_image002

That’s why I proposed to dump all the logs in Azure log analytics to get them indexed and have alerting / reports on them.

It took some convincing to get the SCCM engineers to believe this is possible but it is actually quite simple to set it up using log analytics and custom logs.

So first up the requirements:

  • You need to have an active azure subscription
  • You need to have Log analytics workspace
  • You need to have a SCCM server onboarded on that workspace.

If these are met the following steps will ensure that the custom logs are coming in:

· Select your workspace in the log analytics blade and select “advanced settings”

clip_image004

Navigate to “Data” => Custom Logs => Add +

clip_image006

This opens the 4 step process with is basically all that is to it.

clip_image008

Step 1: Select a sample log file with the format required. Note that this sample logfile can’t exceed a size of 500k

For this I’ve selected a file on my SCCM site server which was called : SMS_CLOUDCONNECTION

clip_image010

Click browse => select the file => upload => click next

clip_image012

Step 2:

Select the record delimiter:

This is a 2 way choice :

  • Either you choose that every line is a new record in Log Analytics
  • You specify a date format

Note : If there’s no date format selected Log analytics will fill the field “date generated” with the date that the logfile was uploaded instead of the alert / log entry occured.

clip_image014

Step 3 : Adding log collection paths:

This is where Log analytics is going to look for the log files.

A couple of things to keep in mind:

  • The path you fill in here will be checked on ALL machines which are onboarded to the Azure Log Analytics workspace
  • If you want a specific log you fill in the full name
  • If you want all logs with a certain extension you can actually use wildcards as well
  • You can add multiple logs to the same custom type.

For demo purposes I’ve added the path to all logfiles in SCCM as shown below and I’m uploading all *.LOG files.

The advantage of using the wildcards is in fact that no logs get missed. If new logfiles are created due to size issues the new logfile will be picked up as well

clip_image016

Step 4 :

Add a name for all the records. This name is actually called a type within Log Analytics. This type will hold all the log entries and will be your first stop to start querying.

clip_image018

Click done and at this point the new custom log has been created. The log analytics agents will get notified and will search for logs in that specific directory.

clip_image020

After a while the logs will be parsed and be available in log analytics to query.

In the next blog post I’ll show how to efficiently search across these types.

Meet the Belgians at MMS DE !

MMS in general, if it’s now MMSMOA or MMSDE, one of the best conferences around for me to attend or speak!
The reason to be there is that their is no better community focused conference available where you could have a direct interaction during the sessions with MVP’s and Product team members. If you know uservoice, well then you have a lot of walking uservoice options as interaction with Product team people like David James (Director of Product Engineering) and that is just priceless!
We Belgians are greatly represented at MMS Desert edition :
  • Opening with MMS 2018 Desert Edition Welcome Reception – That will be hilarious
  • Monitoring Configmgr
  • Patch management with log analytics! Best of both worlds?  
  • Enterprise Mobility Suite Part 1 & 2
But if you really want to go deepdive on technical Configmgr, Modern Management or simply Belgian Beer questions, come to the Belgian Style Cabana sessions hosted on Tuesday , december 4th ! 
Furthermore there are 2 other belgian speakers ,Tom Degreef and Kim Oppalfens , both Enterprise Mobility MVP’s at MMS DE .
See the lineup of other gresat speakers here: https://de.mmsmoa.com/directory/speakers
Are you still doubting to come to the MMS Desert edition next week in December 2-4 with two full conference days? Don’t , if you want to learn a lot, meet up with the Belgian’s and do cabana sessions Belgian style! Register Now at https://mmsmoa.com/desertedition
Hope to meet you there!
Kenny Buntinx
MVP Enterprise Mobility

Enough talk, let’s build
Something together.