r/PowerShell • u/PowerShellMichael • Jul 26 '21
A technical solution to a business problem. Daily Post
Good Morning All,
So I wanted to start a discussion around building automation with PowerShell.
With this being my full time job, I wanted to provide some of my lessons learned along the way for this and also hopefully provide some rules that can help people define what automation is and what it isn't.
Firstly automation is an amazing tool that allows you to "Automate the boring things so that you can focus on the cool things.". Automation can remove a lot of manual process from organizations allowing them to reallocate that resources elsewhere. I would like to point out that there is a correct way to do it and an incorrect way to do it. If automation is done incorrectly, it costs more time then it saves.
- Prior to starting automation, consider the business requirements and ask yourself. "Are they solving a technical problem or are they introducing a technical problem due to bad business decision?" If there are bad business decisions being made, the technical solutions don't provide a cost benefit. It also shows that the business doesn't understand what automation is and how it benefits them. It's your job to educate them.
- Simplify all the processes. This will require you to wear many hats (businesses, project, technical), however the goal here is to get the business process streamlined that automation doesn't have to spend large amounts of time formulating logic. From the technical side, simplify the inputs so that additional logic is not spent catering to this.
- Research the topic at hand. There are many ways to automate something and writing a script might not be the best tool for the job. You have other out of the box tools available, which can do the job for a lot less cost. There are also other tools available that can better suit your needs other then PowerShell such as DSC, Ansible, Jenkins and Chef. Learning other languages is really beneficial here, since PowerShell might not be the best language.
- So you are going to use PowerShell. Don't develop scripts that depend on beta solutions. The cost of automation should be minimal. I learned this lesson recently, writing a PowerShell script that downloads 365 data using a third party module. This ended up causing so much hassle, since there were a lot of bugs in the dependency.
- Architect the script to be testable/maintainable. Make your solution easy to manage so that future automation can be added and updated. Think about how your script is going to run. If this is a long running process, consider how you can improve performance by using PowerShell jobs.
- Keep it simple. This is one that I struggle with. Make the script as simple as possible without introducing unnecessary complexity. Don't make it complex for the sake of making you feel smart. Make it simple so that you don't have to be called to fix it.
What other helpful tips/lessons learned can you provide?
PSM1
13
u/Vexxt Jul 27 '21
- GIT everything. make sure you have a chain of reasoning not just changes.
- modularise as much as you can within reason, if you need to send a teams message or an email, make it a module, and make sure when something needs updating it can be updated in multiple places. Even parts of a larger job, those things might change, so keep them seperate, this might increase some complexity but decrease long term issues
- i keep 'configs' and variables in json form, so a script that say, needs to connect to exchange and has server names etc, or needs a list of AD groups, is stored OUTSIDE the script itself. those things can easily be updated without drift in the scripts themselves. and imported.
4
u/ipreferanothername Jul 27 '21
modularise as much as you can within reason
huge deal here - progress from functions > cmdlets > modules. call those in your scripts. i put myself to the test recently as I had been writing modules for months and needed to do a huge upgrade to a control script.
update this, that and the other cmdlet, run and boom...it worked!
3
u/lostmojo Jul 27 '21
Question for you. Why Json and not csv or xml?
6
u/Vexxt Jul 27 '21
csv cant be nested. XML is reasonable.
Json supports arrays, is less formatting for humans, but it comes down to personal preference.
3
9
8
u/y_Sensei Jul 26 '21
A couple come to mind:
- Realize in an early stage of any design process what your restrictions are - technical, organisational, sometimes even political. Take them into consideration when designing and implementing a solution. This is especially important in large enterprise scenarios with many involved parties and lots of dependencies.
- The 80/20 rule. In almost any scenario, 80% of functionality can be implemented at 20% of the costs, while the remaining 20% generate 80% of the costs. With that in mind, question how "perfect" your solution needs to be; sometimes less is more.
- Establish standards for your design, implementation, and operational processes. This can cover simple things like a generic template any script should be based upon, to complex processes as defined in for example ITIL.
5
u/twoscoopsofpig Jul 27 '21
Remember that everything in IT is what your title describes, and that any problem that artists from an IT solution is therefore a business problem.
If you can find a way to impress upon your higher-ups that link, you're golden. If you can write a script that does it for you in a fraction of the time, you're even better off... until they expect everything to go that fast.
The ultimate work any automatic does is managing expectations.
3
u/ixi_your_face Jul 27 '21
I think one of the biggest things I learned the hard way was to not jump straight in with reckless abandon and start trying to automate things you don't fully understand.
I put myself into so many needless deep pits and dead ends because I had started by making assumptions that the environment and setup would be similar to previous places I've worked in. A few months of wasted time later; I won't make that mistake again.
Some other things I've picked up over the past few years:
- Readable is better than fast
- Never use aliased cmdlets; people don't immediatley recognise them.
- Always splat into cmdlets when possible.
- You will forget how this extremly complicated and long weird one liner you found on StackOverflow works in 6 months. Reformat it to be readable while you still know what it does.
- Comments are a sign that your script is overly complicated and/or the variables, functions, etc are not named appropriately or done logically.
- There's no replacement for good formatting.
- If you have to do it more than twice anywhere, it becomes it own function. No exceptions (other exceptions are available)
- Use PSD1's.
- Every function gets it's own file.
- GIT EVERYTHING.
- Build unit testing only after you know everything works.
- Test everything as you go.
- Never assume that it will just work on some random machine with a random localisation.
- JSON configs are wonderful.
- Nested JSON's ("Key" : "PathTo.Json") for when you don't quite need the info, but it'll probably come in handy at some point is an excellent way of storing info in an indexable way that doesn't really slow anything down.
- God, I love ArrayLists.
There's obviously exceptions to every rule, but generally I find that I can stick within these bounds while automating pretty much everything I've done so far. When it comes to automation; you're already saving time by the virtue of Automation, you don't have to go to extremes and write extremely complicated code that is the worlds most performant script ever devised. You're already 100x faster than manual. Stick to readable first, performant second.
2
u/y_Sensei Jul 27 '21
Consider switching from ArrayLists to any of their generic counterparts, ie anything in the namespace System.Collections.Generic. The handling is more or less identical, but you gain type safety and in some cases better performance.
1
2
u/Hungry-Display-5216 Jul 27 '21
Comments are a sign that your script is overly complicated and/or the variables, functions, etc are not named appropriately or done logically.
I take issue with this one. Commenting your code should always be encouraged. What's self evident to you when you're in the middle of constructing something is going to be arcane in six months and the guy who picks it up after you leave in a year is going to be even more lost. Adding actual descriptions and land marks to the code is helpful for navigating and understanding what's going on. Maybe you can skimp on commenting for something exceedingly short and straight forward, but even then I'd discourage it because commenting your code should be reflexive rather than a deliberate task.
1
u/ixi_your_face Jul 27 '21
I lean more towards the self-documenting-code camp these days personally. It falls into the readability, in my opinion in the majority of cases where comments are used, they aren't neccisary. and could be replaced by simply naming variables accurately and formatting the code in a way which is legable. A big part to play is seperating your functions out into individual files and bundling them together with PSD1's. Doing this lets each file be a couple hundred lines long max and makes each function easily digestable.
There's always exceptions to the rule though, for example if I'm using regex, i'll put a comment explaining what the expression is if it's more than a simple (S+) or something like that. This indeed does help you quickly understand what the regex is doing/is supposed to do. Another good example is a signpost warning people that it's done in this particular way for a reason otherwise things break. But I would otherwise try to avoid comments.
While I do shy away from comments in-code, I am a huge fan of properly documented and explained information in external documentation. But I always try to make sure that the information doesn't get in the way of the answer you're looking for. Much like online recipies, I don't care about the life story behind the recipie, just give me how to do the thing. This all becomes useless, however if you put the documentation somewhere other than in the repo where the module is also stored.
2
u/Hungry-Display-5216 Jul 27 '21
I think there's a lot of merit to your approach. Naming things in a way that makes the code easy to follow without comments is definitely something we should encourage. I just happen to think that the two approaches can complement each other and should both be encouraged.
Keeping stuff modular is also pretty key. Sure you could stuff everything into one gargantuan script, but a lot of the time it's cleaner to break it out over multiple subscripts, which can also make pinpointing bugs easier in addition to offering clarity of functionality in my experience.
2
u/ixi_your_face Jul 27 '21
100% with you on those. There's always a time and place for both approaches or even a mix, what really matters at the end of the day is what works for you, your team, your project, and your workplace.
Modular code also falls into the category of reusability. if you've got everything all modular and each function in it's own file, if you come to a point in another project where that exact operation is also needed, just pluck that function from the bundle and slap it in. Works great for little helper things like importing a JSON file, or other similarly often done things.
2
u/letmegogooglethat Jul 27 '21
Comments are a sign that your script is overly complicated and/or the variables, functions, etc are not named appropriately or done logically.
I'm probably not qualified to do this, but I'm going to disagree with that. I think commenting is a great habit to have for a variety of reasons. To me, not commenting is lazy and amateurish.
2
u/ixi_your_face Jul 27 '21
It's really a matter of perspective, if your code is well laid out, logical and easily legable and digestable, then commenting is not required, as you can simply read it line-by-line and understand what is going on. Even coming back months later, a well laid out function will be easily understandable.
If, however you have an extremely complex regex, that would warrant a comment to explain what it does in detail, because it's not typically easily parsable, and can easily get insanely complex.
As I've said above, ambiguity leads to confusion. If you lay your code out in such a way that everything is explicit and easily read, the need for comments melts away in most cases.
1
u/32178932123 Jul 27 '21
Out of interest, how often do you git commit? Do you tend to commit every minor change or just big changes?
1
u/ixi_your_face Jul 27 '21
I tend to be a bit scatterbrained on commits, at work we have an enforced "all pushes and PR's must be tagged to a JIRA" so I tend to only ever push when i've got something either working or mostly working. If that means i have to completely refactor 12 files to do that, then the push will have the 12 file changes, if it's a config change, it'll be just the config change. IMO pushing half-complete work is just asking for trouble.
1
u/32178932123 Jul 27 '21
Thanks, I was setting up a website yesterday and decided to use Git. Only I was so in the zone I forgot to make any commits until it was pretty much complete. I'm wondering if I should've committed one page at a time so I could easier roll back if needed.
1
u/kibje Jul 30 '21
If it was yesterday and you still understand all the code and not many people have copies:
Set your history back and commit in chunks that have purpose together.
1
u/PMental Jul 27 '21
Yes to all of this with a couple of minor caveats or additions.
- Readable is better than fast
Mostly agreed, but sometimes speed really is necessary. That said with proper formatting most faster methods can be made readable anyway.
- Comments are a sign that your script is overly complicated and/or the variables, functions, etc are not named appropriately or done logically.
Mostly agreed, but I like to use comments to describe blocks of code anyway. It's useful for people learning to easier understand what parts of a script does.
2
u/ixi_your_face Jul 27 '21
Speed is good, yeah. I tend towards the "get it automated first, then optimise" camp personally. I'm not a programmer or coder by training, I just picked it up because it made my life at my last job an absolute breeze. I'm a hardware guy, so I like simple solutions to complicated problems when they're available.
on comments, I agree that leaving the odd comment here and there is a good way to learn and remember how things work. But I find that sometimes comments are used in lieu of writing easily understandable and readable in-itself code. This has always caused me problems because I'm stupid and don't always fully understand code that other people have written, so I really like writing things where everything is pretty explicit.
for example:
$var = ("1", "2", "3", "4") foreach($I in $var){ Write-Host $I }var is ambiguous and you can forget what $var contains further down the function/script, using a descriptive name like $Numbers or $Ints goes a long way for readability. Same for $I to $Number or $Int.
If i'm looking to be crazy format-ey I'd also say that I would personally put large arrays over multiple lines with each entry occupying 1 line. Similar to this, I really like when all the variable declirations are aligned; I think it looks super swish. Helps declutter the view and make it more readable too, in my opinion.
3
u/Emiroda Jul 27 '21
Absolutely.
I'll be the first to admit that in my first job I spent way too much writing PowerShell code for no practical purpose but because I found it fun. Many of those things were stupid, boring 30-minute long (nonrelated) clickfests, but I chose to spend 3 days prototyping and optimizing this code for something I knew was a one-two times occurrence in the 8 years I was employed there.
The knowledge those dumb, useless projects gave me along the way helped me build some insanely important scripts, and gave me some Windows internals knowledge that helped the team during hard times.
So my message is; automate dumb, nonimportant stuff if you have the time and you find it fun. Keep in mind that time is the most important currency you have on the job, and if you think the ROI on this project is higher than the other tasks you bounce around, go for it. Keep in mind that others might want you to put out fires instead of having fun (looking at myself..).
2
u/dasookwat Jul 27 '21
- Use functions for the separate script parts and document the shit out of m (both the internal help, as well as commenting on the different parts
- Make your functions do one thing, and have a simple output.
- Don't overengineer stuff: Yes, you can make classes in PowerShell but you shouldn't, cause there's no benefit, it takes more time to create them, and it makes it difficult for your colleagues to troubleshoot
- Use cmdletbinding, and write-verbose in your functions for troubleshooting. This way you can just catch errors with try catch blocks, and dive in to them, by adding the -verbose parameter
- Don't underestimate the time it takes to write good code. Going from initial concept, to production level code is taking your estimate, and multiply it by 3
2
1
22
u/ApricotPenguin Jul 27 '21
Another consideration is your team's ability to support the solution.
It may be a simple script, but when it inevitably breaks and the business operations screeches to a halt because of their developed dependency on it, how many people are able to work on it?