At this point in time, Your organization most likely uses its website to deliver key business data to your customers.  This could include the delivery of product marketing information, contact information, or product support documentation.  Your product may be your website if you deliver your application in a SaaS or cloud based distribution model.  As a customer, when I pay for something, I am telling that company that I trust they can deliver not only what they sold me as a product, but all necessary services that come with supporting that product.  In other words, I am telling that company that I trust them.  As part of this trust relationship, we expect that the systems that enable this content delivery, such as web servers, are trustworthy, and the integrity of the content has also been ensured.

Organizations may not have to work hard to initially develop this trust relationship with their customer.  What an organization may not be doing is working hard to maintain this trust relationship.  If the integrity of these assets are challenged, there is a potential risk of destroying this trust relationship, which could be cause for a costly recovery attempt of this customer trust, or complete customer loss.  Ensuring the integrity of your organization’s digital assets should be considered a foundational component of any organization’s security practice.

When we speak of digital assets, we not only include the traditional media data types including images, audio, and video, but all other content produced by your company that exists in a persistent state on disk.  In a web server environment we would most likely be dealing the with following types of data:

• Html, Javascript, CSS, and embedded web objects
• PHP, Perl, .NET, and other application code
• Non-HTML Document types such as PDF or Office files
• Images, Audio, and Video

We were recently employed to review the state of a customer’s environment following a recent intrusion.  As with a portion of the environments we work in today, the services provided included web content and application delivery provided by Apache running on Linux.  While Apache does provide access logging for files requested from it, it does not maintain state regarding the integrity of the files it serves. Following an intrusion, identifying the the attack vector is extremely important in providing future security of this environment but should not be the only consideration.

The following questions need to be asked concurrently:

• What organizational or customer data was exposed?

• Was any data modified and what is the potential impact to our customers?

In high traffic environments, it can be extremely difficult to answer both of these questions quickly, which in turn can prolong the delivery of customer communication or notification for external entities that may have a stake in the exposed data.  To speed up the time to derive an answer to both of these questions, there are two methods that are available to expedite this process.

First, the majority of server operating systems in production today have kernel facilitated auditing capability bundled with the operating system.  Linux provides the Linux Audit Subsystem.  Microsoft Windows Server including 2003 and 2008 provide auditing capability.  Solaris, MacOS X, and the BSD family of operating systems also implement audit facilities.  Each of these respective implementations provide the ability to monitor file access and modification events and produce audit trail which can be used to quickly determine which critical assets were accessed or modified.  Although the deployment of auditing policy is not trivial, the benefit can easily be measured if file integrity is violated and you are able to effectively determine the targeted assets and the associated scope of exposure.

Secondly, organizations should deploy a file integrity monitoring system such as Tripwire or Samhain.  These systems utilize one way cryptographic functions (also known as message digest algorithms) such as MD5, SHA1, SHA2, or Tiger, to create a catalog of computed hashes of  files covered by the monitoring software’s defined policy.  Following the a baseline definition process, these systems monitor filesystem changes against prior hash calculations, and in some cases, against known bad hash values associated with exploit, rootkit code, and other potential malware.   When a change event is discovered, notifications can be delivered to those accountable.

The data generated by each of these tools should be streamed via encrypted transport to a centralized syslog server.   This centralized server should exist in a logically distinct network segment from all other nodes in the environment.   Because this server essentially becomes the gold copy record for file integrity in your environment as related to file assets, extreme care should be used to ensure the validity of logs captured.  This includes utilizing file integrity monitoring and limiting access to those who have a need to know.

The employment of these systems do not provide protection against intrusion but can ease the burden of cleaning up the mess and help organizations identify impact.  While this does not completely mitigate potential loss of trust with your customers, it allows you to effectively measure whether or not their trust was violated and the overall level of exposure.  In the future, we will be providing direction on how these logs can be utilized to provide real-time alerting of an attack in progress, and what you can do to decrease your time to react.

Enterprise IT, Web Operations, security | No Comments »

“What do I do about the vulnerable open source components installed in my commercial products?”

I have been asked that question many times, and I wish the reply I had to deliver was one with a better message.  Unfortunately we often find vulnerable components embedded in commercial products.  Vulnerabilities that can lead to the compromise of the system they are installed in.  We have seen this in both software products that companies are actively shipping, as well as in software and appliances other clients have purchased and installed.  When we request that the vulnerable components be upgraded, the change request very rarely makes it very far up the chain.  The reasoning (when one is actually given) is other “feature enhancements” and new functionality for customers and sales are a higher priority.

While performing security audits against organizations this is an issue that comes up frequently.  What we have seen is that unless the client represents a significant source of revenue or other opportunities for the company shipping vulnerable products, getting them to change can be a very difficult and time consuming process that may not yield major results in any workable time frame.  Baring being a significant revenue source, being a potentially good or bad publicity source could also be a way to get them to be a little more helpful.  For instance having a large scale outage caused by an exploit of one of the vulnerably pieces of software they are using.  However I do not really recommend waiting to be exploited as a viable option.

I am not suggesting that you do not open a support ticket with the company when you do find an issue; however receiving an unworried/unconcerned response is something you should expect.  Having a proof of concept attack against their software that you can show them would be one way to help get their attention, but that can be time consuming and costly.

Instead most organizations are forced to mitigate the issue themselves.  While waiting for the software companies to upgrade or fix the issues you have found, you need to mitigate and create controls around all of the known issues.  In general this can be accomplished through a six step process.

1. Segment your network into different zones based upon function, confidentiality and importance of data, etc.
2. Deploy host based and network based firewalls to restrict access to specific ports, sources, and/or destinations.  The firewalls should restrict both what connections can be made in an attempt to exploit a vulnerability, as well as to limit what damage or access a compromised computer can create.  You do not want a compromised server downloading tools or initiating more attacks if you can help it.
3. Deploy and USE an operational Alerting and Monitoring system, configured to detect outages, error conditions, and anomalies, .  This includes traffic flows, service and device uptime, and SLA measurements, as well as the gathering of syslog data and appropriate snmp-traps.
4. Deploy a network and/or host based IDS to watch for any attempts at exploiting known vulnerabilities or traffic anomalies.
5. Deployment of an event correlation system (such as Cisco’s MARS, RSA’s enVision, or Splunk) to help manage the massive amounts of server, firewall, and IDS logs.
6. Continually monitor, maintain, and manage your environment.  There are no easy “place it and forget it” solutions to security issues.

Enterprise IT, security | No Comments »

I recently had a client ask if additional categories could be added to the choice list in the SharePoint 2010 Calendar Web Part.  These categories are in the pick list when you create a new calendar item from the SharePoint interface.

This is how to change the categories:
1. Go to the calendar web part from the browser.
2. Under the Calendar Tools tab, click the Calendar tab.
3. Choose List Settings.
4. Scroll down to the Columns heading.
5. Click Category.
6. Under the Additional Column Settings heading, there is a text box with the categories, and above the box reads: “Type each choice on a separate line.”
7. Add or remove categories, then click OK.

Enterprise IT, Web Operations | No Comments »

The documentation for SharePoint 2010 is gradually being filled in, but one important piece seems to be missing: the comparison of the feature sets for the Server and Foundation Editions. Some organizations are deploying their first SharePoint farm with the 2010 Beta and need this edition information to choose the version to deploy. Looking at the forums, it seems like I’m not the only person who has this question.

If you have any information about the features available in Server versus Foundation, let us know.

Update: 5/12/2010: The official comparison page has been posted: http://sharepoint.microsoft.com/en-us/buy/Pages/Editions-Comparison.aspx

Enterprise IT | 2 Comments »

This error is showing up in quite a few forum posts with no definitive solution. I was able to resolve this for a MOSS 2007 small farm running on Windows Server 2008 64-bit. The main symptom is that when you try to open your search settings in the SSP, you get this message in the browser window:

The search service is currently offline. Visit the Services on Server page in SharePoint Central Administration to verify whether the service is enabled. This might also be because an indexer move is in progress.

In my case, looking in the Windows Application log, I had event ID 10036 messages from Gatherer.  These messages indicated that the search service account did not have access to stored procedures in two of the databases.

The problem resulted from changing the search service account without adding permissions for the new account to the search and SSP databases.  After adding the account permissions in SQL and restarting the osearch service for SharePoint, the search settings in the SSP were available.

During troubleshooting, I also found a set of three event IDs (6398, 6482, 7076) and messages that and Administrative job could not run.  That problem was resolved with this hotfix: http://support.microsoft.com/default.aspx/kb/946517.

Uncategorized | No Comments »

Microsoft has written a step-by-step instructional for setting up a proof of concept lab to demonstrate NAP with 802.1X on the new Windows 2008 NPS. NPS on Windows 2008 replaces IAS on Windows 2003, and new Network Access Protection functionality is now built in. The guide can be downloaded from here: http://www.microsoft.com/downloads/details.aspx?FamilyID=8a0925ee-ee06-4dfb-bba2-07605eff0608&displaylang=en.  The guide is very detailed and easy to follow, but there’s one catch: it’s written for Vista, and there are differences from the way 802.1X authentication works on XP.  I got it working by compiling information from several sources and updating the step-by-step document with the changes.  Here, I will tell you what edits I made, so you can do the same.

On page 19, under Top Level Heading: Install the Group Policy Management feature
The heading below it should read:
“To install the Group Policy Management feature,”
not:
“To install the NPS server role.”

On page 25, under Heading: Verify NAP policies, in the numbered list under “To verify NAP policies”
2. reads:
“Verify that the NAP connection request policy you created in the previous procedure is first in the processing order, or that other policies that match NAP client authentication attempts are disabled. Also verify that the status of this policy is Enabled. The default name of this policy is NAP 802.1X (Wired). ”

Add to that: “Open the policy and navigate to Settings > Authentication Methods.  Make sure Override network policy authentication settings is checked and that under EAP types, Microsoft: Protected EAP (PEAP) is shown.”

In the section starting on page 26, under the Top Level Heading: Configure NAP client setting in Group Policy, under “To configure NAP client settings in Group Policy:”
between steps 12 and 13, insert the following:
13.  In the console tree, navigate to Computer Configuration\Windows Settings\Security Settings\Network Access Protection\NAP Client Configuration\Enforcement Clients.
14.  In the details pane, right-click each enforcement client you want to enable, and then click Enable.
15.  In the console tree, navigate to Computer Configuration\Windows Settings\Security Settings\Wired Network (IEEE 802.3) Policies.
16.  Right-click the Wired Network…and click Create a New Windows Vista Policy.  Name the policy, and make sure Use Wired AutoConfig is checked.
17.  Click on the security tab and Enable IEEE 802.1X… and for Select and network authentication method, select Microsoft: Protected EAP (PEAP).
18.  Click Properties… and make sure Validate server certificate is checked.  Also check Enable Fast Reconnect and Enable Quarantine checks.  Select Authentication Method should show Secured password (EAP-MSCHAP v2).  Click OK.

Side note: As I was troubleshooting, the NPS log in the expanded Windows 2008 Event Viewer was invaluable to tracking down issues.  You no longer have to read IAS format logs for basic troubleshooting.

Uncategorized | No Comments »

I needed to move a SharePoint 2007 front end from a Windows 2003 32-bit to a Windows 2008 64-bit server while leaving the databases on the existing SQL 2005 server. The contents of this article appear many different places and appear to be the Microsoft accepted method for accomplishing the change: http://technet.microsoft.com/en-us/library/dd622865.aspx. If you want to move everything, that’s the way to go. Since I just wanted to move the SharePoint server itself, and I wanted a way to fall back in case there were compatibility issues, I created a staged approach. It consisted of the following major steps:

1. Build up the new Windows 2008 x64 server, install MOSS 2007 SP1 on it, and create a test site.
2. Install all applications and components that are on the original server onto the new server.
3. Plan downtime and migrate one Site Collection.
4. Test the Site Collection, and then record the exact steps that worked best.
5. Migrate the other Site Collections and decommission the 2003 server.

Since there were many steps and tricks, I wanted to share the full process. These are the assumptions about the SharePoint environment for the purposes of the directions: MOSS 2007; Windows 2003 front-end that also hosts the Central Administration Site; backend SQL 2005 server; needed to do a staged migration to ensure smooth transition for production sites; Maintained same SQL 2005 server on back end, but it would have been the same process with a new server.

1. Document your existing installation. Record such items as:

• third-party web parts
• specialized DLLs – make sure there is a version compiled for 64-bit OS
• templates (stsadm -o enumtemplates)
• packages (stsadm -o enumsolutions)
• presence of static paths
• which web applications are linked to which databases

2. Prepare the Windows 2003 server:
Make sure it is at least upgraded to MOSS SP1. If possible, update it to the latest cumulative update. http://technet.microsoft.com/en-us/library/cc263467.aspx
The Sharepoint installer account will need to be a local administrator on the SQL server, and you will need to log into the SharePoint server as that account during the installation process.
3. Prepare the Windows 2008 x64 sever:
a. Use these instructions to install MOSS 2007 on the new server: http://technet.microsoft.com/en-us/library/cc287748.aspx
b. Add any web parts or other specialized components recorded in Step 1.
c. Configure the permissions.
d. Configure the SSP. It is theoretically possible to migrate an SSP, but I found the procedure to be more trouble than comparing the two side-by-side and replicating the setttings.
4. Perform a test site migration:
a. Make a SQL backup of the content database.
b. Create a blank database with a new database name.
c. Restore the backup into the new database.
d. Create a web application on the new server, and specify the new database name during the creation process.
e. Check the site collection administrators to make sure you are there.
f. If required, do an IIS reset (“iisreset /noforce” at the command line).
g. If using a host header for the site (intranet.company.com), create a DNS entry pointing to the new server with a test site name (intranettest.company.com).
5. After testing of the migration is complete, perform the production migration:
a. Notify users that there will be some downtime.
b. Check that no timer jobs are running.
c. Quiesce the farm for five minutes.
d. Run the preparetomove command for your content database.
e. Make a SQL backup of the content database.
f. Restore the SQL backup over the top of the test database for the new farm.
g. In Central Administration, remove and re-add the content database to the web application.
h. IIS reset.
i. Test internal and external (if applicable) access to the site. Also do some functionality checks: alerts, search (after a full crawl), navigation (static links). Check the Windows event logs for errors.
6. Cleanup:
a. Remove the web application and IIS site from the original farm.
b. Remove the SharePoint installer account from the local administrators on the SQL server.
c. Remove the DNS entry for the testing site.
7. Back up your new environment as soon as it is in a satisfactory state.

One final note: Since I was using a fully qualified domain name for the site name, and I wanted to check functionality of the site from the local server, I ran into the Loopback Check security feature, in which Windows 2008 blocks requests coming from the local machine to prevent reflection attacks. This resulted in a 401 error. As explained here, do not simply set disableloopbackcheck = 1 to get around this. Instead, browse the site from another machine, or use Method 1 from this Microsoft article, in which you specify the host names that should be allowed locally.

Uncategorized | 1 Comment »

Enterprise IT, security | No Comments »

I needed to upgrade MOSS 2007 farm on a Windows 2003 Server from the RTM version to Service Pack 1 as a required step to prepare for a migration to Windows 2008 64-bit.  The process involved installing the WSS 3.0 SP1, ignoring the Configuration Wizard, running the MOSS 2007 SP1, then running the Configuration Wizard.  In this case, the  last step of the Configuration Wizard failed.   The Wizard screen said to look in the event log, which showed three errors similar to this:

Configuration of SharePoint Products and Technologies failed.  Configuration must be performed in order for this product to operate properly.  To diagnose the problem, review the extended error information located at C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\12\LOGS\PSCDiagnostics_5_8_2009_19…log, fix the problem, and run this configuration wizard again.

I consulted the PSCDiagnostics log, but all it did was repeat the error message shown above.  Looking in the Upgrade.log file, the last entry looked like this:

[SPManager] [INFO] [5/8/2009 7:45:10 PM]: Inplace Upgrade session finishes. root object = SPFarm Name=SharePoint_Config, recursive = True. 1 errors and 0 warnings encountered.

Searching back up through the file, I found one area where errors were reported:

[SPManager] [ERROR] [5/8/2009 7:45:03 PM]: Pre-Upgrade [SPSite Url=http://sitename/sites/TestSite] failed. Microsoft.SharePoint.Upgrade.SPSiteWssSequence has the ContinueOnFailiure bit set. Moving on to the next object in sequence.
[SPManager] [ERROR] [5/8/2009 7:45:03 PM]: The system cannot find the path specified. (Exception from HRESULT: 0×80070003)

The error was caused by a site that is no longer there.  This happens sometimes when a site is deleted from SharePoint, but it doesn’t get removed all the way from the configuration database.  It will often show up in Central Administration after you have deleted it.

Looking through the Microsoft SharePoint Service Pack 1 installation documentation, there were no fixes in the troubleshooting section that fit this scenario.  I found many articles about removing orphaned sites, but this site was not a true orphan (its parent was not missing).  If you have a true orphan, you can use the stsadm parameter databaserepair.

In this case, the solution was to detach the content database within SharePoint then reattach it.  When you do that, it removes the old entry from the configuration database.  This operation can normally be done by detaching and reattaching a content database within Central Administration, but since the upgrade was not complete and the farm disabled, I used stsadm.

stsadm –o deletecontentdb –url http://sitename –database contentdatabasename –databaseserver sqlserver\sqlinstance
stsadm –o addcontentdb –url http://sitename –database contentdatabasename –databaseserver sqlserver\sqlinstance

*The deletecontentdb parameter does not delete the database, it only detaches the reference to it from your SharePoint farm.

After this, I ran psconfig at the command line:

psconfig -cmd upgrade -inplace b2b -wait -force

Once that completed successfully, I launched the Configuration Wizard manually from All Programs > Administrative Tools > SharePoint Products and Technologies Configuration Wizard to get visual confirmation that the upgrade was completed.

After upgrading SharePoint, always check the Upgrade.log in %COMMONPROGRAMFILES%\Microsoft Shared\Web server extensions\12\LOGS for “Finished upgrading SPFarmn Name=<configuration database>”, “0 errors and 0 warnings” at the end.

Enterprise IT | 2 Comments »

Let me start by saying this:   I am not a fan of CoyotePoint load balancers.    My support experiences so far have all been atrocious.   The system architecture is a cheap imitation of F5′s BigIP architecture from a decade ago which constantly limits me.    I’m convinced that people only buy these things because they’re cheap.

I’ve been working with a customer who’s exceeded the throughput capabilities of their Equalizer E350si load balancer.   Although the marketing materials will tell you that this unit is capable of throughput up to 700Mbps (hah!),  we were maxing out and dropping packets above 50 Mbps.

Rant:  You see, the problem lies in Coyote’s system architecture.  This E350si is powered by a NetBurst-architecture Pentium 4 2.8Ghz, with HyperThreading disabled.   Coyote uses a FreeBSD-4 based kernel, which was well known for it’s slow timers, slow interrupt handling, and immature device polling implementation.  In this classic system architecture, each incoming packet generates an Interrupt Request (IRQ), which must be serviced by the CPU in a time-slotted fashion.   So what we have is a load balancer which reports it’s CPU as being mostly idle,  but in reality cannot handle packets quickly enough.   THIS IS NOT HOW YOU DESIGN NETWORK EQUIPMENT, PEOPLE.   End rant.

Good news:   In version 8, Coyote introduced a new mode of operation called DSR, or Direct Server Return.  DSR is quite clever, really, because it get’s around Coyote’s packet handling limitation (to a big degree) by handling the incoming network packets, but allowing the web servers to respond directly to clients.   This cuts the number of TCP packets the Coyote has to process in half,  and cuts the number of Ethernet frames by much more if you consider that the return packets are much larger.

Here’s how it works.   In a traditional setup,  the Coyote receives a packet on its external interface (em1), makes a load balancing decision, and then forwards to the packet along to a host behind its internal interface (em0).   Most shops NAT here as well, for security and/or IP address conservation reasons.   So the Coyote must perform Layer 2 – 4 (or 7) processing of the packet as it receives it,   then make a load balancing decision,  then translate the packet (that’s the T in NAT),  then re-process the packet going out the internal interface.   Then, rinse, lather, repeat for the return packet.   Such is the life of a typical load balancer.

In DSR mode, you start by chopping off the Internal interface of the load balancer altogether and eliminating NAT.   This requires moving your webservers onto publicly routable IP addresses, so please make sure they are firewalled properly.   Now you have your load balancer and webservers all on the same ethernet segment.   You create a VIP (Virtual IP) on the load balancer, and then add that SAME VIP address as a loopback address to the webservers!

You’re probably scratching your head, wondering how this is going to work.  I know that I was.  Here’s the magical part.   Only the load balancer responds to ARP requests for the VIP.  The webservers have Apache listening on the VIP address,  but don’t respond to ARP requests at all on that address.   Each incoming packet is sent from the router to the MAC address of the load balancer,  which then makes a load balancing decision and then sends an identical copy of that packet to the MAC address of the web server.   Let me say that again.  The load balancer performs no more translation, it literally just copies the packet over to the webserver.   Since the source MAC address is unchanged,  the web server replies directly to the router and skips the load balancer entirely.

Sounds a bit scary, but works well.  Except for one thing.   In their brilliance,  the Coyote folks created a section in the Manual with configuration instructions for “Linux/Unix Systems”, but ACTUALLY put in instructions for BSD-like systems only.  Who runs FreeBSD anymore?   DON’T TRY THESE INSTRUCTIONS ON A LINUX SERVER UNLESS YOU WANT TO LOCK YOURSELF OUT.

On Linux,  the correct way to create the loopback address is by adding a “labelled” loopback interface,  but ALWAYS set the netmask of your new interface to “255.255.255.255″.   If you match the netmask of the VIP,  your webserver will stop responding to packets on it’s external interface.  Very bad.

So,  assume your public VIP address is 12.34.56.78  (fake, to protect the innocent),  and your webserver’s address is 12.34.56.80.   Create a loopback address like so:

/sbin/ifconfig lo:vip inet 12.34.56.78 netmask 255.255.255.255

Then, the output of “ifconfig -a” looks something like this:

eth0      Link encap:Ethernet  HWaddr 00:40:A4:8E:B0:1A
inet addr:12.34.56.80  Bcast:12.34.56.255  Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
RX packets:47131905 errors:0 dropped:0 overruns:0 frame:0
TX packets:77804088 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:5111837334 (4875.0 Mb)  TX bytes:104047655003 (99227.5 Mb)
Interrupt:177

lo        Link encap:Local Loopback
inet addr:127.0.0.1  Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING  MTU:16436  Metric:1
RX packets:1311164 errors:0 dropped:0 overruns:0 frame:0
TX packets:1311164 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:439975560 (419.5 Mb)  TX bytes:439975560 (419.5 Mb)

lo:vip    Link encap:Local Loopback
inet addr:12.34.56.78  Mask:255.255.255.255
UP LOOPBACK RUNNING  MTU:16436  Metric:1

If it all works, you should be able to confirm correct operation by using tcpdump or Wireshark/Ethereal on the webserver, and verifying that the SOURCE address is your VIP address and you’re seeing lots of 200 OK messages.

tshark -n -i eth0 -R http.response port 80

Enterprise IT, Web Operations | No Comments »