Organizations may not have to work hard to initially develop this trust relationship with their customer.  What an organization may not be doing is working hard to maintain this trust relationship.  If the integrity of these assets are challenged, there is a potential risk of destroying this trust relationship, which could be cause for a costly recovery attempt of this customer trust, or complete customer loss.  Ensuring the integrity of your organization’s digital assets should be considered a foundational component of any organization’s security practice.

When we speak of digital assets, we not only include the traditional media data types including images, audio, and video, but all other content produced by your company that exists in a persistent state on disk.  In a web server environment we would most likely be dealing the with following types of data:

• Html, Javascript, CSS, and embedded web objects
• PHP, Perl, .NET, and other application code
• Non-HTML Document types such as PDF or Office files
• Images, Audio, and Video

We were recently employed to review the state of a customer’s environment following a recent intrusion.  As with a portion of the environments we work in today, the services provided included web content and application delivery provided by Apache running on Linux.  While Apache does provide access logging for files requested from it, it does not maintain state regarding the integrity of the files it serves. Following an intrusion, identifying the the attack vector is extremely important in providing future security of this environment but should not be the only consideration.

The following questions need to be asked concurrently:

• What organizational or customer data was exposed?

• Was any data modified and what is the potential impact to our customers?

In high traffic environments, it can be extremely difficult to answer both of these questions quickly, which in turn can prolong the delivery of customer communication or notification for external entities that may have a stake in the exposed data.  To speed up the time to derive an answer to both of these questions, there are two methods that are available to expedite this process.

First, the majority of server operating systems in production today have kernel facilitated auditing capability bundled with the operating system.  Linux provides the Linux Audit Subsystem.  Microsoft Windows Server including 2003 and 2008 provide auditing capability.  Solaris, MacOS X, and the BSD family of operating systems also implement audit facilities.  Each of these respective implementations provide the ability to monitor file access and modification events and produce audit trail which can be used to quickly determine which critical assets were accessed or modified.  Although the deployment of auditing policy is not trivial, the benefit can easily be measured if file integrity is violated and you are able to effectively determine the targeted assets and the associated scope of exposure.

Secondly, organizations should deploy a file integrity monitoring system such as Tripwire or Samhain.  These systems utilize one way cryptographic functions (also known as message digest algorithms) such as MD5, SHA1, SHA2, or Tiger, to create a catalog of computed hashes of  files covered by the monitoring software’s defined policy.  Following the a baseline definition process, these systems monitor filesystem changes against prior hash calculations, and in some cases, against known bad hash values associated with exploit, rootkit code, and other potential malware.   When a change event is discovered, notifications can be delivered to those accountable.

The data generated by each of these tools should be streamed via encrypted transport to a centralized syslog server.   This centralized server should exist in a logically distinct network segment from all other nodes in the environment.   Because this server essentially becomes the gold copy record for file integrity in your environment as related to file assets, extreme care should be used to ensure the validity of logs captured.  This includes utilizing file integrity monitoring and limiting access to those who have a need to know.

The employment of these systems do not provide protection against intrusion but can ease the burden of cleaning up the mess and help organizations identify impact.  While this does not completely mitigate potential loss of trust with your customers, it allows you to effectively measure whether or not their trust was violated and the overall level of exposure.  In the future, we will be providing direction on how these logs can be utilized to provide real-time alerting of an attack in progress, and what you can do to decrease your time to react.

I have been asked that question many times, and I wish the reply I had to deliver was one with a better message.  Unfortunately we often find vulnerable components embedded in commercial products.  Vulnerabilities that can lead to the compromise of the system they are installed in.  We have seen this in both software products that companies are actively shipping, as well as in software and appliances other clients have purchased and installed.  When we request that the vulnerable components be upgraded, the change request very rarely makes it very far up the chain.  The reasoning (when one is actually given) is other “feature enhancements” and new functionality for customers and sales are a higher priority.

While performing security audits against organizations this is an issue that comes up frequently.  What we have seen is that unless the client represents a significant source of revenue or other opportunities for the company shipping vulnerable products, getting them to change can be a very difficult and time consuming process that may not yield major results in any workable time frame.  Baring being a significant revenue source, being a potentially good or bad publicity source could also be a way to get them to be a little more helpful.  For instance having a large scale outage caused by an exploit of one of the vulnerably pieces of software they are using.  However I do not really recommend waiting to be exploited as a viable option.

I am not suggesting that you do not open a support ticket with the company when you do find an issue; however receiving an unworried/unconcerned response is something you should expect.  Having a proof of concept attack against their software that you can show them would be one way to help get their attention, but that can be time consuming and costly.

Instead most organizations are forced to mitigate the issue themselves.  While waiting for the software companies to upgrade or fix the issues you have found, you need to mitigate and create controls around all of the known issues.  In general this can be accomplished through a six step process.

1. Segment your network into different zones based upon function, confidentiality and importance of data, etc.
2. Deploy host based and network based firewalls to restrict access to specific ports, sources, and/or destinations.  The firewalls should restrict both what connections can be made in an attempt to exploit a vulnerability, as well as to limit what damage or access a compromised computer can create.  You do not want a compromised server downloading tools or initiating more attacks if you can help it.
3. Deploy and USE an operational Alerting and Monitoring system, configured to detect outages, error conditions, and anomalies, .  This includes traffic flows, service and device uptime, and SLA measurements, as well as the gathering of syslog data and appropriate snmp-traps.
4. Deploy a network and/or host based IDS to watch for any attempts at exploiting known vulnerabilities or traffic anomalies.
5. Deployment of an event correlation system (such as Cisco’s MARS, RSA’s enVision, or Splunk) to help manage the massive amounts of server, firewall, and IDS logs.
6. Continually monitor, maintain, and manage your environment.  There are no easy “place it and forget it” solutions to security issues.

This is how to change the categories:
1. Go to the calendar web part from the browser.
2. Under the Calendar Tools tab, click the Calendar tab.
3. Choose List Settings.
4. Scroll down to the Columns heading.
5. Click Category.
6. Under the Additional Column Settings heading, there is a text box with the categories, and above the box reads: “Type each choice on a separate line.”
7. Add or remove categories, then click OK.

If you have any information about the features available in Server versus Foundation, let us know.

Update: 5/12/2010: The official comparison page has been posted: http://sharepoint.microsoft.com/en-us/buy/Pages/Editions-Comparison.aspx

Configuration of SharePoint Products and Technologies failed.  Configuration must be performed in order for this product to operate properly.  To diagnose the problem, review the extended error information located at C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\12\LOGS\PSCDiagnostics_5_8_2009_19…log, fix the problem, and run this configuration wizard again.

I consulted the PSCDiagnostics log, but all it did was repeat the error message shown above.  Looking in the Upgrade.log file, the last entry looked like this:

[SPManager] [INFO] [5/8/2009 7:45:10 PM]: Inplace Upgrade session finishes. root object = SPFarm Name=SharePoint_Config, recursive = True. 1 errors and 0 warnings encountered.

Searching back up through the file, I found one area where errors were reported:

[SPManager] [ERROR] [5/8/2009 7:45:03 PM]: Pre-Upgrade [SPSite Url=http://sitename/sites/TestSite] failed. Microsoft.SharePoint.Upgrade.SPSiteWssSequence has the ContinueOnFailiure bit set. Moving on to the next object in sequence.
[SPManager] [ERROR] [5/8/2009 7:45:03 PM]: The system cannot find the path specified. (Exception from HRESULT: 0×80070003)

The error was caused by a site that is no longer there.  This happens sometimes when a site is deleted from SharePoint, but it doesn’t get removed all the way from the configuration database.  It will often show up in Central Administration after you have deleted it.

Looking through the Microsoft SharePoint Service Pack 1 installation documentation, there were no fixes in the troubleshooting section that fit this scenario.  I found many articles about removing orphaned sites, but this site was not a true orphan (its parent was not missing).  If you have a true orphan, you can use the stsadm parameter databaserepair.

In this case, the solution was to detach the content database within SharePoint then reattach it.  When you do that, it removes the old entry from the configuration database.  This operation can normally be done by detaching and reattaching a content database within Central Administration, but since the upgrade was not complete and the farm disabled, I used stsadm.

stsadm –o deletecontentdb –url http://sitename –database contentdatabasename –databaseserver sqlserver\sqlinstance
stsadm –o addcontentdb –url http://sitename –database contentdatabasename –databaseserver sqlserver\sqlinstance

*The deletecontentdb parameter does not delete the database, it only detaches the reference to it from your SharePoint farm.

After this, I ran psconfig at the command line:

psconfig -cmd upgrade -inplace b2b -wait -force

Once that completed successfully, I launched the Configuration Wizard manually from All Programs > Administrative Tools > SharePoint Products and Technologies Configuration Wizard to get visual confirmation that the upgrade was completed.

After upgrading SharePoint, always check the Upgrade.log in %COMMONPROGRAMFILES%\Microsoft Shared\Web server extensions\12\LOGS for “Finished upgrading SPFarmn Name=<configuration database>”, “0 errors and 0 warnings” at the end.

I’ve been working with a customer who’s exceeded the throughput capabilities of their Equalizer E350si load balancer.   Although the marketing materials will tell you that this unit is capable of throughput up to 700Mbps (hah!),  we were maxing out and dropping packets above 50 Mbps.

Rant:  You see, the problem lies in Coyote’s system architecture.  This E350si is powered by a NetBurst-architecture Pentium 4 2.8Ghz, with HyperThreading disabled.   Coyote uses a FreeBSD-4 based kernel, which was well known for it’s slow timers, slow interrupt handling, and immature device polling implementation.  In this classic system architecture, each incoming packet generates an Interrupt Request (IRQ), which must be serviced by the CPU in a time-slotted fashion.   So what we have is a load balancer which reports it’s CPU as being mostly idle,  but in reality cannot handle packets quickly enough.   THIS IS NOT HOW YOU DESIGN NETWORK EQUIPMENT, PEOPLE.   End rant.

Good news:   In version 8, Coyote introduced a new mode of operation called DSR, or Direct Server Return.  DSR is quite clever, really, because it get’s around Coyote’s packet handling limitation (to a big degree) by handling the incoming network packets, but allowing the web servers to respond directly to clients.   This cuts the number of TCP packets the Coyote has to process in half,  and cuts the number of Ethernet frames by much more if you consider that the return packets are much larger.

Here’s how it works.   In a traditional setup,  the Coyote receives a packet on its external interface (em1), makes a load balancing decision, and then forwards to the packet along to a host behind its internal interface (em0).   Most shops NAT here as well, for security and/or IP address conservation reasons.   So the Coyote must perform Layer 2 – 4 (or 7) processing of the packet as it receives it,   then make a load balancing decision,  then translate the packet (that’s the T in NAT),  then re-process the packet going out the internal interface.   Then, rinse, lather, repeat for the return packet.   Such is the life of a typical load balancer.

In DSR mode, you start by chopping off the Internal interface of the load balancer altogether and eliminating NAT.   This requires moving your webservers onto publicly routable IP addresses, so please make sure they are firewalled properly.   Now you have your load balancer and webservers all on the same ethernet segment.   You create a VIP (Virtual IP) on the load balancer, and then add that SAME VIP address as a loopback address to the webservers!

You’re probably scratching your head, wondering how this is going to work.  I know that I was.  Here’s the magical part.   Only the load balancer responds to ARP requests for the VIP.  The webservers have Apache listening on the VIP address,  but don’t respond to ARP requests at all on that address.   Each incoming packet is sent from the router to the MAC address of the load balancer,  which then makes a load balancing decision and then sends an identical copy of that packet to the MAC address of the web server.   Let me say that again.  The load balancer performs no more translation, it literally just copies the packet over to the webserver.   Since the source MAC address is unchanged,  the web server replies directly to the router and skips the load balancer entirely.

Sounds a bit scary, but works well.  Except for one thing.   In their brilliance,  the Coyote folks created a section in the Manual with configuration instructions for “Linux/Unix Systems”, but ACTUALLY put in instructions for BSD-like systems only.  Who runs FreeBSD anymore?   DON’T TRY THESE INSTRUCTIONS ON A LINUX SERVER UNLESS YOU WANT TO LOCK YOURSELF OUT.

On Linux,  the correct way to create the loopback address is by adding a “labelled” loopback interface,  but ALWAYS set the netmask of your new interface to “255.255.255.255″.   If you match the netmask of the VIP,  your webserver will stop responding to packets on it’s external interface.  Very bad.

So,  assume your public VIP address is 12.34.56.78  (fake, to protect the innocent),  and your webserver’s address is 12.34.56.80.   Create a loopback address like so:

/sbin/ifconfig lo:vip inet 12.34.56.78 netmask 255.255.255.255

Then, the output of “ifconfig -a” looks something like this:

eth0      Link encap:Ethernet  HWaddr 00:40:A4:8E:B0:1A
inet addr:12.34.56.80  Bcast:12.34.56.255  Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
RX packets:47131905 errors:0 dropped:0 overruns:0 frame:0
TX packets:77804088 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:5111837334 (4875.0 Mb)  TX bytes:104047655003 (99227.5 Mb)
Interrupt:177

lo        Link encap:Local Loopback
inet addr:127.0.0.1  Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING  MTU:16436  Metric:1
RX packets:1311164 errors:0 dropped:0 overruns:0 frame:0
TX packets:1311164 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:439975560 (419.5 Mb)  TX bytes:439975560 (419.5 Mb)

lo:vip    Link encap:Local Loopback
inet addr:12.34.56.78  Mask:255.255.255.255
UP LOOPBACK RUNNING  MTU:16436  Metric:1

If it all works, you should be able to confirm correct operation by using tcpdump or Wireshark/Ethereal on the webserver, and verifying that the SOURCE address is your VIP address and you’re seeing lots of 200 OK messages.

tshark -n -i eth0 -R http.response port 80

Unfortunately, WordPress isn’t setup for high performance operation out of the box.  Each page request is very CPU and database intensive.

Under benchmarking, we discovered that each of the customer’s HP DL385 servers could serve between 5-10 WordPress page view’s per second, depending on the page.  (For the pedantic,  I’m considering anything that hits PHP as a page view.)  And this is AFTER I had put major effort into MySQL performance tuning.    Something had to be done, 5 page views per second is just not going to cut the mustard.

In comes the WP-Super-Cache plugin to save the day.   WP-Super-Cache is a plugin which seems like it should be installed with every WordPress instance by default.   It writes out entire pages to static .html files,  and then instructs Apache to serve up the static .html files directly (using mod_rewrite),  therefore avoiding any CPU-gobbling calls to PHP or the database.   But WP-Super-Cache is smart,  it automatically expires cached pages when the content is updated  (by the author, or via comments).

As a result,  we went from 5-10 page views per second to between 500-2000 theoretical page views per second.   At this point we were hitting bandwidth bottlenecks,  which is where I like to be.   As long as webservers can serve up enough data to fill their own pipe, you have happy system administrators (and UNhappy network administrators).

In the past, I’ve warned Windows shops to use unique local Administrator passwords wherever possible.  I’ve even proven the dangers of using the same local Administrator password during a penetration test in 2007.   Combine this with the fact that  I rarely have anything polite to say about VBscript (it’s not a pretty language to work with), and we have the perfect karmic storm.

Yours truly, coding in VBscript, tasked with setting a unique, strong passwords on each one of a few hundred machines.   Here’s what I came up with:

' ChangeLocalAdminOnServers.vbs
' Created by Irving Popovetsky (irving@prostructure)
' 12/15/2008, ProStructure Consulting
'
' Warning:  This script will begin changing passwords as soon
' as it collects a complete list of machine names.
'
' Read and understand this code carefully before executing,
' and always remember to fill in your own variables where appropriate.
' We assume no liability for damages that may be caused by running
' this code in your production environment!!

On Error Resume Next

Dim fso, MyFile
Set fso = CreateObject("Scripting.FileSystemObject")

' ***CHANGEME*** Change the output file to a location you trust, like an
' Encrypted folder or USB stick that can be stored away
' In the future, this could be improved to output directly to PGP or equivalent.
Set MyFile = fso.CreateTextFile("c:\Temp\Changedservers.txt", True)

Const ADS_SCOPE_SUBTREE = 2

Set objConnection = CreateObject("ADODB.Connection")
Set objCommand =   CreateObject("ADODB.Command")
objConnection.Provider = "ADsDSOObject"
objConnection.Open "Active Directory Provider"
Set objCommand.ActiveConnection = objConnection

objCommand.Properties("Page Size") = 1000
objCommand.Properties("Searchscope") = ADS_SCOPE_SUBTREE 

' ***CHANGEME*** Fill in your own Domain name here
objCommand.CommandText = _
    "SELECT Name FROM 'LDAP://dc=DOMAIN,dc=INTERNAL' WHERE objectCategory='computer'"
Set objRecordSet = objCommand.Execute

objRecordSet.MoveFirst

Do Until objRecordSet.EOF
    strComputer = objRecordSet.Fields("Name").Value

	' ***CHANGEME*** Skip the Domain Controllers  - fill in your own values here
	if Instr(1,strComputer, "DOMAINCONTROLLER1") Then objRecordSet.MoveNext
	if Instr(1,strComputer, "DOMAINCONTROLLER2") Then objRecordSet.MoveNext
	if Instr(1,strComputer, "DOMAINCONTROLLER3") Then objRecordSet.MoveNext

	' Irving - Random password
	Dim intUpperLimit, intLowerLimit, strPassword
	strPassword = ""
	intUpperLimit = 126
	intLowerLimit = 33

	For i = 1 to 12
	    Randomize
	    intASCIIValue = Int(((intUpperLimit - intLowerLimit + 1) * Rnd) _
	        + intLowerLimit)
	    strPassword = strPassword & Chr(intASCIIValue)
	Next

	'  Perform the Action.  Write out the computername/password then execute
	MyFile.WriteLine(strComputer & "   " & strPassword)
	Set objUser = GetObject("WinNT://" & strComputer & "/Administrator")
	objUser.SetPassword strPassword

    objRecordSet.MoveNext
Loop

MyFile.Close

Credits to The Scripting Guy’s article on scripting the change of the local administrator password. Very funny article,  I’m a big fan of the Scripting Guy.

Now, there are certainly some improvements that can be made, and WILL be made if I ever have to use this thing again.   First off, the ability to define the output location and LDAP search path.   Second, automatically determining if a server is a domain controller and skipping it.  You DEFINITELY DO NOT want this script hitting a Domain Controller, because it will change the Domain’s Administrator account, and that can be a bad thing.  Trust me, I already learned that lesson, at least I had the password in my output file.

Three articles on the StorageMojo blog are brilliant and worth a read or a revisit:

• NetApp Weighs In On Disks – Val Bercovici of Netapp weighs in on disk failure surveys from Google and CMU’s Parallel Data lab.  Beware some marketing spin mixed in with some real gems of knowledge.
• Google’s Disk Failure Experience:  Google’s analysis of failure rates among 100,000 drives.  It’s worth noting that Google uses cheap SATA drives almost exclusively in the cloud.
• Everything You Know About Disks Is Wrong: The aforementioned CMU study, which has a nice comparison between cheapo SATA disks and fancy FC disks.

There are some critically important takeways in these articles and the surrounding commentary that we must never forget when safeguarding our data.   Most imporant points:

• MTBF is a nearly irrelevant number. Storage experts are contending that Mean Time Beteween Failures is actually much closer between “consumer” SATA drives and “enterprise” FC drives than the marketing people want you to believe.
• There is a huge amount of magic and complexity happening inside of every hard drive: Really, almost all of it is masked from users.   Hard drive controllers and their respective firmware has gotten insanely complex to keep up the increasing number of failure and error scenarios as disks get more dense.   The difference between two disk firmware revisions or code branches can make a considerable impact on drive reliability.
• The REAL difference between the low and high-end is RAID controller and drive firmware smarts: Why do disks and RAID controllers form storage vendors cost so much more?   Because those storage vendors are on the hook for a complete product, and put engineering time into changing the disk behaviour by customizing drive firmware and pre-qualifying drives.   They also know that SMART is a sham, predicting only a small fraction of disk failures.  The secret sauce is performing advanced failure profiling on the RAID controller, and coordinating it with fully understood and tuned drive firmware.  This is the true difference between your cheapo Promise variety RAID setups and those from Netapp, Hitachi, EMC and HP.
• Drive failure rates do go up with age: Heavily used drives are either going to fail in the first three months,  or with steadily increasing risk beyond the 3 year mark.    We’ve seen this in the field as well,  even early $1000/drive 300GB FC disks had a 10% failure rate in the first 3 months after you started working them hard.   Drives older than 3 years have typically been spinning, without stopping, for that entire time period.   Bad Things are known to happen when you spin them down, let them cool, and then try to spin them up again.
• Background media scanning is the best way to detect drive failures: Does your RAID controller perform background media scanning or at least a full array consistency check on a regular basis?  Great!   Because THIS is the real way to predict disk failure,  by monitoring and trending subtle disk failures (bad sectors, etc),  rather than waiting for SMART to (not) predict a major mechanical failure.   As it was said above,  SMART is mostly useless.   Hopefully you’re not feeling quite so good about that “S.M.A.R.T. Status OK” message anymore.
• RAID 5 is more harmful than you think: Many numbers are starting to come out about how often double failures are happening in RAID5 arrays.  It’s quite disturbing, but you run a HUGE risk of data loss during that high-activity array rebuild after your first disk has failed.   And, as pointed out, too many people rely on RAID5 as a backup solution.  So let’s say it again RAID5 is NOT a backup solution, and never will be.