Recently, I got an excellent chance to put my money where my mouth is.

In the past, I’ve warned Windows shops to use unique local Administrator passwords wherever possible.  I’ve even proven the dangers of using the same local Administrator password during a penetration test in 2007.   Combine this with the fact that  I rarely have anything polite to say about VBscript (it’s not a pretty language to work with), and we have the perfect karmic storm.

Yours truly, coding in VBscript, tasked with setting a unique, strong passwords on each one of a few hundred machines.   Here’s what I came up with:

' ChangeLocalAdminOnServers.vbs
' Created by Irving Popovetsky (irving@prostructure)
' 12/15/2008, ProStructure Consulting
'
' Warning:  This script will begin changing passwords as soon
' as it collects a complete list of machine names.
'
' Read and understand this code carefully before executing,
' and always remember to fill in your own variables where appropriate.
' We assume no liability for damages that may be caused by running
' this code in your production environment!!

On Error Resume Next

Dim fso, MyFile
Set fso = CreateObject("Scripting.FileSystemObject")

' ***CHANGEME*** Change the output file to a location you trust, like an
' Encrypted folder or USB stick that can be stored away
' In the future, this could be improved to output directly to PGP or equivalent.
Set MyFile = fso.CreateTextFile("c:\Temp\Changedservers.txt", True)

Const ADS_SCOPE_SUBTREE = 2

Set objConnection = CreateObject("ADODB.Connection")
Set objCommand =   CreateObject("ADODB.Command")
objConnection.Provider = "ADsDSOObject"
objConnection.Open "Active Directory Provider"
Set objCommand.ActiveConnection = objConnection

objCommand.Properties("Page Size") = 1000
objCommand.Properties("Searchscope") = ADS_SCOPE_SUBTREE 

' ***CHANGEME*** Fill in your own Domain name here
objCommand.CommandText = _
    "SELECT Name FROM 'LDAP://dc=DOMAIN,dc=INTERNAL' WHERE objectCategory='computer'"
Set objRecordSet = objCommand.Execute

objRecordSet.MoveFirst

Do Until objRecordSet.EOF
    strComputer = objRecordSet.Fields("Name").Value

	' ***CHANGEME*** Skip the Domain Controllers  - fill in your own values here
	if Instr(1,strComputer, "DOMAINCONTROLLER1") Then objRecordSet.MoveNext
	if Instr(1,strComputer, "DOMAINCONTROLLER2") Then objRecordSet.MoveNext
	if Instr(1,strComputer, "DOMAINCONTROLLER3") Then objRecordSet.MoveNext

	' Irving - Random password
	Dim intUpperLimit, intLowerLimit, strPassword
	strPassword = ""
	intUpperLimit = 126
	intLowerLimit = 33

	For i = 1 to 12
	    Randomize
	    intASCIIValue = Int(((intUpperLimit - intLowerLimit + 1) * Rnd) _
	        + intLowerLimit)
	    strPassword = strPassword & Chr(intASCIIValue)
	Next

	'  Perform the Action.  Write out the computername/password then execute
	MyFile.WriteLine(strComputer & "   " & strPassword)
	Set objUser = GetObject("WinNT://" & strComputer & "/Administrator")
	objUser.SetPassword strPassword

    objRecordSet.MoveNext
Loop

MyFile.Close

Credits to The Scripting Guy’s article on scripting the change of the local administrator password. Very funny article,  I’m a big fan of the Scripting Guy.

Now, there are certainly some improvements that can be made, and WILL be made if I ever have to use this thing again.   First off, the ability to define the output location and LDAP search path.   Second, automatically determining if a server is a domain controller and skipping it.  You DEFINITELY DO NOT want this script hitting a Domain Controller, because it will change the Domain’s Administrator account, and that can be a bad thing.  Trust me, I already learned that lesson, at least I had the password in my output file.

Enterprise IT, security | No Comments »

I was recently interviewed by Michelle Rafter for Inc. Technology about best practices for Administrative Passwords.

Article link:  Psst! Whats the Password?

Announcements, security | No Comments »

ProStructure Consulting is pleased to announce the successful conclusion of its IV&V project for the Washington State Ferries Wireless High-Speed Data Network, and the publication of its full report on the website of the Federal Transit Authority!

Earlier this year, ProStructure completed a major IV&V (Independent Verification & Validation) project for the Washington State Ferries, a department of WSDOT, and the Federal Transit Authority, a department of USDOT.  The project focused on a new high speed ship-to-shore wireless network implemented on the Ferries system by contractor Mobilisa.

The full report is available here:  http://www.fta.dot.gov/documents/CAI-PSC-IVandV-Final-Report-20080910.pdf

From the FTA research site: http://www.fta.dot.gov/assistance/research/research_5638.html

I’m personally excited about the public release of this report for several reasons.   First and most importantly, we get to show the world the kind of truly first-rate work that we do.   Reports of this caliber are almost always locked away as “Highly Confidential” and never see the light of day.  Thanks to the FTA for ensuring that all of its research reports are made publicly available.

In addition to that, we get to show off some of the really innovate technology we put together for this project.  We built a monitoring platform for this project that was both weather hardened and security hardened.   Our locked, tamper evident, IP67 rated (waterproof) boxes were designed to be strapped to the side of the ferries like a flight recorder or “black box”,  continually running a suite of network tests against its twin unit located at the ferry terminal.

Our yellow Test & Measurement Device aboard a WSF vessel

Irving Popovetsky aboard the Steilacoom II performing a final once-over of the unit before the test

By utilizing the scientific method, we build and tested our hypotheses about the expected network performance.  We collected a great deal of addition information including radio statistics, GPS data and weather data.  Using this information we were able to understand how various external influences affected network performance.

For more details, check out the full report.

Announcements | No Comments »

As a quick refresher before I make some recommendations to a customer,  I got to revisit the discussion of hard drives and how they like to fail.

Three articles on the StorageMojo blog are brilliant and worth a read or a revisit:

• NetApp Weighs In On Disks - Val Bercovici of Netapp weighs in on disk failure surveys from Google and CMU’s Parallel Data lab.  Beware some marketing spin mixed in with some real gems of knowledge.
• Google’s Disk Failure Experience:  Google’s analysis of failure rates among 100,000 drives.  It’s worth noting that Google uses cheap SATA drives almost exclusively in the cloud.
• Everything You Know About Disks Is Wrong: The aforementioned CMU study, which has a nice comparison between cheapo SATA disks and fancy FC disks.

There are some critically important takeways in these articles and the surrounding commentary that we must never forget when safeguarding our data.   Most imporant points:

• MTBF is a nearly irrelevant number. Storage experts are contending that Mean Time Beteween Failures is actually much closer between “consumer” SATA drives and “enterprise” FC drives than the marketing people want you to believe.
• There is a huge amount of magic and complexity happening inside of every hard drive: Really, almost all of it is masked from users.   Hard drive controllers and their respective firmware has gotten insanely complex to keep up the increasing number of failure and error scenarios as disks get more dense.   The difference between two disk firmware revisions or code branches can make a considerable impact on drive reliability.
• The REAL difference between the low and high-end is RAID controller and drive firmware smarts: Why do disks and RAID controllers form storage vendors cost so much more?   Because those storage vendors are on the hook for a complete product, and put engineering time into changing the disk behaviour by customizing drive firmware and pre-qualifying drives.   They also know that SMART is a sham, predicting only a small fraction of disk failures.  The secret sauce is performing advanced failure profiling on the RAID controller, and coordinating it with fully understood and tuned drive firmware.  This is the true difference between your cheapo Promise variety RAID setups and those from Netapp, Hitachi, EMC and HP.
• Drive failure rates do go up with age: Heavily used drives are either going to fail in the first three months,  or with steadily increasing risk beyond the 3 year mark.    We’ve seen this in the field as well,  even early $1000/drive 300GB FC disks had a 10% failure rate in the first 3 months after you started working them hard.   Drives older than 3 years have typically been spinning, without stopping, for that entire time period.   Bad Things are known to happen when you spin them down, let them cool, and then try to spin them up again.
• Background media scanning is the best way to detect drive failures: Does your RAID controller perform background media scanning or at least a full array consistency check on a regular basis?  Great!   Because THIS is the real way to predict disk failure,  by monitoring and trending subtle disk failures (bad sectors, etc),  rather than waiting for SMART to (not) predict a major mechanical failure.   As it was said above,  SMART is mostly useless.   Hopefully you’re not feeling quite so good about that “S.M.A.R.T. Status OK” message anymore.
• RAID 5 is more harmful than you think: Many numbers are starting to come out about how often double failures are happening in RAID5 arrays.  It’s quite disturbing, but you run a HUGE risk of data loss during that high-activity array rebuild after your first disk has failed.   And, as pointed out, too many people rely on RAID5 as a backup solution.  So let’s say it again RAID5 is NOT a backup solution, and never will be.

Enterprise IT | No Comments »

Microsoft has released a security advisory (954462) warning users that automated SQL injection attacks against ASP and ASP.NET based sites are escalating.  Unlike about 5 years ago, most web developers I talk to today understand what SQL injection is and how to defend against it.

The new automated attacks are mostly focused on Content Management System (CMS) functionality, in order to insert an IFRAME on your website pointing to a malicious site.  5 years ago we all talked about how SQL injection attacks would focus on stealing credit card numbers and personal information directly out of databases, or skipping password checks or adding administrative users.   I demonstrated that last one myself during a security engagement.   The attacks were jaw-dropping and scary, and we thought that everyone was on the way to getting it fixed.

If you find this XKCD comic funny, you’re halfway toward understanding exactly how dangerous SQL injection is:

So why are ASP and ASP.NET still such easy pickings?   I believe it’s due to two main reasons:

1.  ASP has THE WORST string validation functionality of any modern programming language used to create web applications today.

Have you ever seen a successful, tight form validation routine written in ASP or VB?   It’s ugly, and easy to get wrong.    This is because there’s no regex (regular expression) library for performing string-fu,  a luxury that pretty much everyone else has.

5-6 years ago, smart developers realized this and quickly said “Hey, we need to filter this garbage before it ever gets to the app, by building ISAPI filters in C and C++”.   Some shops wrote their own, many adopted tools like Microsoft’s URLScan or Eye’s SecureIIS.   Problems were solved, as these ISAPI filters magically denied all sorts of nasty strings, and were totally worth the pain of implementation.

2.  Microsoft left Windows 2003/IIS 6 users out in the cold

Right around the time when Windows 2003 shipped, two things were happening.  First, people like myself were finding MAJOR functionality flaws in URLScan which Microsoft was basically refusing to fix, and telling users to wait for the next version.    Second, instead of actually shipping that next version, Microsoft proclaimed that IIS 6 had “URLScan technology” built in, and nobody needed URLscan anymore.  Too bad that whatever was actually included with IIS 6 was missing URLScan’s best functionality, including the malicious string denial functionality.  Thanks for that one, Microsoft.

So for about 5 years, the memory of URLScan faded, and everyone migrated to Windows 2003 and IIS 6.  Sure, IIS 6 is a lot more “secure by default”, but ASP shops weren’t being adequately protected from SQL injection attacks.    It wasn’t until THIS MONTH that Microsoft released URLScan 3.0 Beta which finally supported IIS 6.   URLScan 3.0 Beta also supports IIS 7, which now has basic request filtering, but not as advanced as URLScan.   It is not too late to consider this tool.

3.  Some ASP and ASP.NET shops are still running insecure code and using insecure coding practices

Microsoft and various security firms have been preaching advice about how to protect against SQL injection attacks for years now.  It is a lack of risk awareness and training that is allowing it to continue.   The argument that it is just too expensive for a business to fix these vulnerabilities doesn’t stand anymore, the attacks are too damaging.

What can be done?

The answer is simple.  You need to self-assess your applications, or find someone to perform the assessment for you.   Be careful, though, self-assessment has a huge risk, which is a golden rule of security:  The designer of a system is the least likely to see its flaws.   Fortunately, if you go this route there are plenty of free and intuitive tools to help you along the way.

HP has released a cool new tool called Scrawlr (free download) that will walk your ASP/ASP.NET/MS SQL based website and search for basic SQL Injection vulnerabilities.   Unfortunately the tool does have some serious limitations  (doesn’t handle authentication, won’t check forms or anything involving a POST, and cannot perform blind SQL injection).   But it is a nicely packaged tool for finding the simplest vulnerabilities, the ones that the bad guys are hitting as well.

Fortunately, there are plenty of other free tools that perform Blind SQL injection (check google),  as well as WebScarab and even the Firefox add-ons TamperData and Hackbar to assist you in self-assessing your own site.

security | No Comments »

Early this year, I pulled the Windows Server 2008 Feature Components and Active Directory posters out of TechNet Magazine and hung them up over my desk. I then excitedly installed Windows Server 2008 on my test server using my TechNet subscription. I was eager to try out the many enhanced features of Microsoft’s new server OS before they reached wide use. One of the most promising features added to this server OS was built-in virtualization (available on x64 systems only). In order to do my daily troubleshooting and research, I need several OS instances and builds available, such as Windows Server 2003, a separate server with MS SQL installed, and SUSE Linux running a nagios monitoring server. Other than Hyper-V, I was also interested to try out the improvements made to Active Directory, such as the restartable AD service.

The first thing I noticed after building up the new server was the Server Manager screen that greets the administrative user on logon. I was impressed with the array of features that can be managed from this console as well as the breadth of the information about each feature that was presented. For instance, after installing DNS, the DNS service information and relevant event logs appear under the DNS feature in the Server Manager console.

The next thing that I noticed was that the new server would not reboot – it would simply hang on shutdown at a black screen interminably. At first, I thought this was the long shut down time that I was warned about. Apparently, there have been improvements made to the shutdown process that reduce the chances of file corruption due to services that fail to shut down completely before the allotted shutdown timeout period.

After a little research, I discovered that several devices drivers could not be found, including the one for ACPI. This meant that I had to be physically present to restart the computer after software updates and OS configuration changes. The test server that I’m using is an off-brand grey box, whereas major manufacturers like HP and Dell have probably already worked out any major driver kinks by this time. However, driver support is something to keep in mind.

After configuring a DNS server on the 2008 box, I installed Active Directory. As promised, the AD process runs as a service that can be stopped and restarted. This is convenient if you need to perform offline operations on a DC. As I explored around the AD tools, I noticed that GPMC was installed by default, so I didn’t have to download and install a toolkit to use it.

Once Active Directory was installed and running, I turned to the biggest new feature in Server 2008 – Hyper-V virtualization. Because it was early in the release cycle, only Hyper-V RC0 was available. Right away it was apparent that it was a 1.0 technology. Hyper-V only supported a few operating systems, and getting those few installed was far more laborious than on the other virtualization system that I’ve used. Many people have expressed interest in this technology, as it promises to save licensing money on virtualization. Stay tuned for an upcoming entry on Hyper-V, because the topic deserves a separate treatment. In the meantime, I recommend holding off for some basic features that are still missing.

Shortly after I had my test server configured just the way I needed to be productive, my activation grace period ran out. I attempted to activate my instance of Windows, but I received a message that the DNS server could not be contacted. I thought this was strange, since the server was a DNS server, and it could contact Microsoft without a problem. Once the grace period expires, basic Windows functions are no longer available, such as updating device drivers automatically. I could have reset the activation period for another 240 days, but this would only be putting off the inevitable.

After extensive searching, I found a Microsoft article that described the problem: “Activation Error: Code 0×8007232b DNS Name does not exist.” According to this article, my options were: a) set up a Key Management Server (KMS), b) use a Multiple Activation Key (MAK), or c) use the license key. Since this was a TechNet-distributed version of Windows, the TechNet site insisted that I did not need a key, and no key was made available. This only left the option of creating a KMS. To prepare for this, I began reading the extensive “Volume Activation 2.0 Resources for Windows Vista and Windows Server 2008.” It was here I discovered that until five instances of Vista and Server 2008 on unique hardware on your network made activation requests of the KMS, no requests would be approved. Here I was stuck, since I did not have five unique computers on which I could install Vista or 2008, just to get my test server activated. It seems that small businesses using TechNet for testing fell through the cracks of Microsoft’s new licensing plan.

After six months of using Windows Server 2008 on my testing server, I concluded that the OS is not mature enough to be worth the trouble of keeping it going in my situation. I do encourage others to try it out in a testing environment to familiarize themselves with some of the nascent features. I’ve rebuilt my test server with Windows Server 2003, and I’ll continue testing Windows Server 2008 as a virtual machine. I look forward to exploring and writing about the greatly improved performance logging capabilities, the new application virtualization feature in Terminal Services, and Hyper-V. Check back for more articles on these topics in the coming weeks. Meanwhile, the Server 2008 posters will continue to hang optimistically over my monitor.

Enterprise IT, Reviews | 4 Comments »

Last week, quite a few major vulnerabilities were discovered in some of our customer’s favorite products, namely:

• VMWare (all products, from ESX Server all the way down to VMware Player)
• Cisco PIX and ASA (versions 7.1, 7.2, 8.0 and 8.1)
• Mac OS X (Both Server and Client editions, 10.4 and 10.5 are affected)

Quite a few of these vulnerabilities are remotely exploitable and especially dangerous on the PIX and unprotected OSX and VMware installations.   VMware also looks like it may have a  local “VM breakout” bug or two, watch out for these.   We strongly recommend getting these products updated as soon as possible.

For more information and relevant links, check out the US-CERT Cyber Security Bulletin SB08-161.  Search for the product you’re running on this page.

Advisories, security | No Comments »

I switched ProStructure over to Zimbra about a month ago.   The good news is that we are absolutely loving it.   The sad part is that this is our 4th open source based workgroup/calendaring solution in 5 years of doing business.

The Backstory

We started out with Webcal.pl for 2 years, then “upgraded” to OpenXchange (which was terrible) for 6 months or so, then SugarCRM for the last 2 years or so.   Nobody was particularly happy with SugarCRM either, the interface was slow and buggy, and the interoperability with fat clients and devices never really worked out.  You could say that our commitment to open source was hurting us, because none of the solutions were successful business enablers.

The hype around Zimbra has been considerable.  Zimbra seemed to be advertised as the best open source based MS Exchange competitor. Although the Zimbra folks mention Exchange as little as possible, it is clearly where they set their sights.

Mini Review

Held at face value, the single most compelling feature for users is Zimbra’s Outlook-killer web interface.  The user experience is really fantastic, I’ve seen Outlook, Thunderbird and Apple Mail/iCal users drop their fat clients and switch to using the web interface exclusively.  It is just that good.   You really have to try it out for yourself to see cool things like the Gmail-style email conversation views, smart tags and calendar views.  The built-in IM functionality is a brilliant add-on as well.

But that’s not all.  The commercial version of Zimbra (which is worth every penny) provides an Outlook connector which provides an Exchange-like user experience that is nearly imperceptible to most users.  You can also purchase (as an add-on) over-the-air sync functionality for Windows Mobile, Palm and Blackberry devices (BES required).

Zimbra is also totally interoperable.  The web interface works under IE, Firefox and Safari.  Zimbra provides classic POP/IMAP/SMTP services for those that want to keep their old mail client.  Calendars can be accessed via the iCalendar protocol, and even exposed to other users as authenticated or open icalendar or free/busy feeds. The IM functionality is fully standards-based, and supports any XMPP capable client like Pidgin/Gaim, Adium or Trillian.

But is Zimbra right for you?

Every business needs Calendaring just as much as it needs Email capability these days.  In my opinion, business fall into 4 camps:

1. You have already invested in a recent version of Exchange or equivalent (Groupwise, Domino, etc)

Probably not.  You’ve already invested 5 or 6 digits worth of capital, if not more, to get to your existing platform.  If it’s running poorly, or seeing low rates of user acceptance, you may want to look at other factors first.  There could be architecture or user expectation issues that could be managed with outside help, if you need it.  You can consider a forklift to a better platform several years down the line, but a wholesale email/calendaring platform switch now may cost you your job.

2. You are running older versions of Exchange (5.5 or 2000)

Lets face it, most IT systems are fully depreciated within 3-5 years.  Microsoft has left you out in the cold with mainstream support.  Your server hardware is ancient and dying.  Or did you try to just move that aging email system to newer hardware?   You know who you are.

Upgrading to newer version of Exchange may not have to be a foregone conclusion for you.  The cost savings of migrating to Zimbra may be significant.  You’ll have to buy new servers, but those servers may be spec’ed a little bit leaner than you’d need for Exchange.  You’ll need to retrain your admins anyway.  The big issue is:  Are you prepared to start running Linux servers?   If you’re a blue-blooded Microsoft shop with absolutely no Linux experience or aspirations, Zimbra is probably not for you, Linux (or MacOS X server) is a requirement.   If you think that you can train (or hire) some Linux talent, and pull it off, you will be loved by your users and management alike.

3. Those that currently run an open source based system (Postfix, Sendmail, etc + Sugar, webcal,etc)

What are you waiting for?  Seriously, you’re probably holding your users back from having proper calendaring and scheduling, AntiSpam and AntiVirus filtering (unless you’re running AMaViS) or internal IM functionality.  Go forth, and download the “Network Edition trial”.   If you decide that the $399 “starter edition” or $35/user pricing is too expensive, let your license expire and it will automatically fall back to the open source edition functionality, disabling Outlook/mobile integration.

4. Those that currently have a hosted/outsourced solution

If letting someone else run your Email server doesn’t give you the heebeejeebies, check out the long list of providers lining up to do hosted Zimbra, which tends to run cheaper per-mailbox than hosted Exchange.

My Only concern about Zimbra is…

Yahoo and Microsoft.   Last year, Zimbra was acquired by Yahoo.  So far, Yahoo has let Zimba flourish, but all of this may change if Microsoft gets its hands on Yahoo.  We thought the threat had passed, but recent news shows that Carl Icahn is launching a proxy battle to replace Yahoo’s board with a more Microsoft-friendly one.

What do you think Microsoft is going to do if it purchases the new single biggest threat to its very profitable Exchange product?  I don’t think anyone who’s been following Microsoft’s decades of brutal predatory takeovers and monopolistic business practices is expecting Zimbra, its staff and any patent portfolio to survive in one piece or be spun off.

The fact that large parts of Zimbra is released under the GPL may not help.   Microsoft may try threaten patent lawsuits against Zimbra’s customers as well as any business that sprouts up to support the GPLed version.  They’ve done it with Linux, remember that they funded Baystar which funded SCO’s lawsuit against IBM, Novell, and a few large Linux customers.

Just to be clear, I’m not saying that the threat of Microsoft takeover should ruin the the taste for Zimbra.  Just be smart, and keep an eye out for the writing on the wall.

-Irving Popovetsky

Principal Consultant, ProStructure Consulting

Enterprise IT, Reviews | No Comments »

Welcome, readers!

Here at ProStructure Consulting, we’re extremely enthusiastic about technology and the trials and tribulations of its implementation in the field.  We are fortunate enough to get to build, manage and secure some of the most sophisticated IT environments in the Pacific Northwest.  As a result of our position, we get to do things that nobody has ever done (or fully documented) before.

We believe that the web in general is and always will be the ultimate resource for IT professionals who are trying to understand, implement and troubleshoot pretty much anything.  The purpose of this blog is to chronicle some of our adventures, document challenges, lessons learned and best practices that nobody has yet written about, and in general give back to the IT community by providing our own resource.

We’re starting out with a team of three writers, each with very unique talents and specialties:

• Irving Popovetsky, Principal Consultant: Anything and everything about information security, systems engineering, storage, disaster recovery, IT policy and management.
• Brandon Psmythe, Principal Consultant: Large-scale networks and monitoring, network engineering, information security, IT policy and management
• Amber Pham, Systems Engineering Consultant: Microsoft Operating Systems and Servers, Project Management, Data Center design and engineering

Stay tuned as we begin to fill out some recent war stories.

Announcements | No Comments »