The brute is back.  By the brute, I mean the SSH brute force attack.  In the last month we have noticed an upturn in SSH brute force traffic targeting hosts addressed in public IPv4 address space.  The highest ranking attack sources we have seen include address allocations in KRNIC, APNIC, and RIPE.  In the cases we’ve encountered, the target user was either the root user or something out of a dictionary.  An SSH brute force attack challenges the complexity of the user generated password associated with a user account by iterating through a dictionary to guess user passwords.  Brute-force code will also generate attempts randomly.  The risk associated with the vulnerability exposed here is great, so its important that some action be taken by administrators to ensure a secure operating environment moving forward.

Does your organization currently employ a strong password policy?  Does your organization currently enforce this strong password policy? A strong password should be 14 characters long and should utilize numeric, upper and lower case alphabetical, and symbol characters such as “!@#$%^&*()”.    Most every modern operating system or directory service (LDAP) implementation supports enforcement of strong password complexity.  Administrators should ensure they are utilizing the password history function of their directory service if available, to minimize the potential for password reuse by end users.  For Linux users not utilizing a directory, you can utilize the PAM module provided by the Openwall project, passwdqc, which provides password complexity enforcement.

If your environment allows, utilize password-less authentication using public key encryption provided by either the RSA or DSS.  Most SSH clients and servers support one or both of these methods.  Finally, Ensure that your SSH server does not allow root or Administrator logins.

If your environment already has a firewall in place to facilitate network access control, you should ensure that TCP port 22 is only available to those networks that need access.  If your environment does not have a firewall, OS level packet filtering is available in all modern operating systems.  When thinking about the SSH surface area in your environment, ensure you include devices that are manageable via SSH including routers, firewalls, switches, and other network appliances.

You should really ask yourself why your SSH service is available to the public if you aren’t explicitly providing a shell service to end users.

security | No Comments »

While there was some security documentation for SharePoint 2007, it was general in nature, and it required browsing around several different documents and pages.  Microsoft has done us a service with the SharePoint 2010 security hardening documentation that was released around the time the product hit RTM.  This documentation includes a secure server snapshot of the services required, and it includes a definitive list of necessary ports for each component.  This is a big win for administrators who need to protect the SharePoint server(s) in an isolated network.

The documentation is divided into two parts, web and SQL, and together they provide the big picture for a secure environment.  I’ve recently used the documentation to design the security for a SharePoint farm that needs to provide access to multiple outside agencies/partners.  It was much easier to use this documentation than what was provided for the previous version.  I did find a few areas where the documentation could have been more clear, so I wanted to share my findings and see if anyone else has feedback to make the recommendations stronger.

Web Front-End

I found that in a standard Windows network, NetBIOS over TCP/IP could be disabled, according to Microsoft’s recommendation.  The article did not include instructions, but I did find a response on TechNet that describes how it’s done.

The web.config settings could have been described a little more clearly.

• The PageParserPaths should be empty by default.
• Remember that whenever you create a new web application, a new web.config file is created for it, so you will need to verify the settings are still secured.
• I couldn’t find any information about this one, and I’d love it if someone who knows could share their thoughts on how to accomplish this suggestion: “Ensure that Web Part limits around maximum controls per zone is set low.”  Where is this set, and what would be considered low?

I wish this documentation had listed the minimum required permissions for each service, as I’m having to discover these myself.  For instance, the web analytics service is the one that writes diagnostic logs, and that service account needs access to write to the diagnostic logs directory.  It would be great to see a definitive list beyond what was offered in the service accounts documentation.

SQL Back-End

Unless you are using named instances on SQL, it is safe to block access to port UDP/1434.  One measure that the documentation did not mention but is critical to protecting SQL servers in general is that the firewall rules should use a default deny for all inbound access to the SQL servers.  Only the application server(s), Domain Controllers, and the workstations of the DBAs should be able to reach the SQL servers at all.

Please share any other insights you might have or other resources that can help us secure our SharePoint environments better.

Uncategorized | No Comments »

At this point in time, Your organization most likely uses its website to deliver key business data to your customers.  This could include the delivery of product marketing information, contact information, or product support documentation.  Your product may be your website if you deliver your application in a SaaS or cloud based distribution model.  As a customer, when I pay for something, I am telling that company that I trust they can deliver not only what they sold me as a product, but all necessary services that come with supporting that product.  In other words, I am telling that company that I trust them.  As part of this trust relationship, we expect that the systems that enable this content delivery, such as web servers, are trustworthy, and the integrity of the content has also been ensured.

Organizations may not have to work hard to initially develop this trust relationship with their customer.  What an organization may not be doing is working hard to maintain this trust relationship.  If the integrity of these assets are challenged, there is a potential risk of destroying this trust relationship, which could be cause for a costly recovery attempt of this customer trust, or complete customer loss.  Ensuring the integrity of your organization’s digital assets should be considered a foundational component of any organization’s security practice.

When we speak of digital assets, we not only include the traditional media data types including images, audio, and video, but all other content produced by your company that exists in a persistent state on disk.  In a web server environment we would most likely be dealing the with following types of data:

• Html, Javascript, CSS, and embedded web objects
• PHP, Perl, .NET, and other application code
• Non-HTML Document types such as PDF or Office files
• Images, Audio, and Video

We were recently employed to review the state of a customer’s environment following a recent intrusion.  As with a portion of the environments we work in today, the services provided included web content and application delivery provided by Apache running on Linux.  While Apache does provide access logging for files requested from it, it does not maintain state regarding the integrity of the files it serves. Following an intrusion, identifying the the attack vector is extremely important in providing future security of this environment but should not be the only consideration.

The following questions need to be asked concurrently:

• What organizational or customer data was exposed?

• Was any data modified and what is the potential impact to our customers?

In high traffic environments, it can be extremely difficult to answer both of these questions quickly, which in turn can prolong the delivery of customer communication or notification for external entities that may have a stake in the exposed data.  To speed up the time to derive an answer to both of these questions, there are two methods that are available to expedite this process.

First, the majority of server operating systems in production today have kernel facilitated auditing capability bundled with the operating system.  Linux provides the Linux Audit Subsystem.  Microsoft Windows Server including 2003 and 2008 provide auditing capability.  Solaris, MacOS X, and the BSD family of operating systems also implement audit facilities.  Each of these respective implementations provide the ability to monitor file access and modification events and produce audit trail which can be used to quickly determine which critical assets were accessed or modified.  Although the deployment of auditing policy is not trivial, the benefit can easily be measured if file integrity is violated and you are able to effectively determine the targeted assets and the associated scope of exposure.

Secondly, organizations should deploy a file integrity monitoring system such as Tripwire or Samhain.  These systems utilize one way cryptographic functions (also known as message digest algorithms) such as MD5, SHA1, SHA2, or Tiger, to create a catalog of computed hashes of  files covered by the monitoring software’s defined policy.  Following the a baseline definition process, these systems monitor filesystem changes against prior hash calculations, and in some cases, against known bad hash values associated with exploit, rootkit code, and other potential malware.   When a change event is discovered, notifications can be delivered to those accountable.

The data generated by each of these tools should be streamed via encrypted transport to a centralized syslog server.   This centralized server should exist in a logically distinct network segment from all other nodes in the environment.   Because this server essentially becomes the gold copy record for file integrity in your environment as related to file assets, extreme care should be used to ensure the validity of logs captured.  This includes utilizing file integrity monitoring and limiting access to those who have a need to know.

The employment of these systems do not provide protection against intrusion but can ease the burden of cleaning up the mess and help organizations identify impact.  While this does not completely mitigate potential loss of trust with your customers, it allows you to effectively measure whether or not their trust was violated and the overall level of exposure.  In the future, we will be providing direction on how these logs can be utilized to provide real-time alerting of an attack in progress, and what you can do to decrease your time to react.

Enterprise IT, Web Operations, security | No Comments »

“What do I do about the vulnerable open source components installed in my commercial products?”

I have been asked that question many times, and I wish the reply I had to deliver was one with a better message.  Unfortunately we often find vulnerable components embedded in commercial products.  Vulnerabilities that can lead to the compromise of the system they are installed in.  We have seen this in both software products that companies are actively shipping, as well as in software and appliances other clients have purchased and installed.  When we request that the vulnerable components be upgraded, the change request very rarely makes it very far up the chain.  The reasoning (when one is actually given) is other “feature enhancements” and new functionality for customers and sales are a higher priority.

While performing security audits against organizations this is an issue that comes up frequently.  What we have seen is that unless the client represents a significant source of revenue or other opportunities for the company shipping vulnerable products, getting them to change can be a very difficult and time consuming process that may not yield major results in any workable time frame.  Baring being a significant revenue source, being a potentially good or bad publicity source could also be a way to get them to be a little more helpful.  For instance having a large scale outage caused by an exploit of one of the vulnerably pieces of software they are using.  However I do not really recommend waiting to be exploited as a viable option.

I am not suggesting that you do not open a support ticket with the company when you do find an issue; however receiving an unworried/unconcerned response is something you should expect.  Having a proof of concept attack against their software that you can show them would be one way to help get their attention, but that can be time consuming and costly.

Instead most organizations are forced to mitigate the issue themselves.  While waiting for the software companies to upgrade or fix the issues you have found, you need to mitigate and create controls around all of the known issues.  In general this can be accomplished through a six step process.

1. Segment your network into different zones based upon function, confidentiality and importance of data, etc.
2. Deploy host based and network based firewalls to restrict access to specific ports, sources, and/or destinations.  The firewalls should restrict both what connections can be made in an attempt to exploit a vulnerability, as well as to limit what damage or access a compromised computer can create.  You do not want a compromised server downloading tools or initiating more attacks if you can help it.
3. Deploy and USE an operational Alerting and Monitoring system, configured to detect outages, error conditions, and anomalies, .  This includes traffic flows, service and device uptime, and SLA measurements, as well as the gathering of syslog data and appropriate snmp-traps.
4. Deploy a network and/or host based IDS to watch for any attempts at exploiting known vulnerabilities or traffic anomalies.
5. Deployment of an event correlation system (such as Cisco’s MARS, RSA’s enVision, or Splunk) to help manage the massive amounts of server, firewall, and IDS logs.
6. Continually monitor, maintain, and manage your environment.  There are no easy “place it and forget it” solutions to security issues.

Enterprise IT, security | No Comments »

I recently had a client ask if additional categories could be added to the choice list in the SharePoint 2010 Calendar Web Part.  These categories are in the pick list when you create a new calendar item from the SharePoint interface.

This is how to change the categories:
1. Go to the calendar web part from the browser.
2. Under the Calendar Tools tab, click the Calendar tab.
3. Choose List Settings.
4. Scroll down to the Columns heading.
5. Click Category.
6. Under the Additional Column Settings heading, there is a text box with the categories, and above the box reads: “Type each choice on a separate line.”
7. Add or remove categories, then click OK.

Enterprise IT, Web Operations | No Comments »

The documentation for SharePoint 2010 is gradually being filled in, but one important piece seems to be missing: the comparison of the feature sets for the Server and Foundation Editions. Some organizations are deploying their first SharePoint farm with the 2010 Beta and need this edition information to choose the version to deploy. Looking at the forums, it seems like I’m not the only person who has this question.

If you have any information about the features available in Server versus Foundation, let us know.

Update: 5/12/2010: The official comparison page has been posted: http://sharepoint.microsoft.com/en-us/buy/Pages/Editions-Comparison.aspx

Enterprise IT | 1 Comment »

This error is showing up in quite a few forum posts with no definitive solution. I was able to resolve this for a MOSS 2007 small farm running on Windows Server 2008 64-bit. The main symptom is that when you try to open your search settings in the SSP, you get this message in the browser window:

The search service is currently offline. Visit the Services on Server page in SharePoint Central Administration to verify whether the service is enabled. This might also be because an indexer move is in progress.

In my case, looking in the Windows Application log, I had event ID 10036 messages from Gatherer.  These messages indicated that the search service account did not have access to stored procedures in two of the databases.

The problem resulted from changing the search service account without adding permissions for the new account to the search and SSP databases.  After adding the account permissions in SQL and restarting the osearch service for SharePoint, the search settings in the SSP were available.

During troubleshooting, I also found a set of three event IDs (6398, 6482, 7076) and messages that and Administrative job could not run.  That problem was resolved with this hotfix: http://support.microsoft.com/default.aspx/kb/946517.

Uncategorized | No Comments »

Microsoft has written a step-by-step instructional for setting up a proof of concept lab to demonstrate NAP with 802.1X on the new Windows 2008 NPS. NPS on Windows 2008 replaces IAS on Windows 2003, and new Network Access Protection functionality is now built in. The guide can be downloaded from here: http://www.microsoft.com/downloads/details.aspx?FamilyID=8a0925ee-ee06-4dfb-bba2-07605eff0608&displaylang=en.  The guide is very detailed and easy to follow, but there’s one catch: it’s written for Vista, and there are differences from the way 802.1X authentication works on XP.  I got it working by compiling information from several sources and updating the step-by-step document with the changes.  Here, I will tell you what edits I made, so you can do the same.

On page 19, under Top Level Heading: Install the Group Policy Management feature
The heading below it should read:
“To install the Group Policy Management feature,”
not:
“To install the NPS server role.”

On page 25, under Heading: Verify NAP policies, in the numbered list under “To verify NAP policies”
2. reads:
“Verify that the NAP connection request policy you created in the previous procedure is first in the processing order, or that other policies that match NAP client authentication attempts are disabled. Also verify that the status of this policy is Enabled. The default name of this policy is NAP 802.1X (Wired). ”

Add to that: “Open the policy and navigate to Settings > Authentication Methods.  Make sure Override network policy authentication settings is checked and that under EAP types, Microsoft: Protected EAP (PEAP) is shown.”

In the section starting on page 26, under the Top Level Heading: Configure NAP client setting in Group Policy, under “To configure NAP client settings in Group Policy:”
between steps 12 and 13, insert the following:
13.  In the console tree, navigate to Computer Configuration\Windows Settings\Security Settings\Network Access Protection\NAP Client Configuration\Enforcement Clients.
14.  In the details pane, right-click each enforcement client you want to enable, and then click Enable.
15.  In the console tree, navigate to Computer Configuration\Windows Settings\Security Settings\Wired Network (IEEE 802.3) Policies.
16.  Right-click the Wired Network…and click Create a New Windows Vista Policy.  Name the policy, and make sure Use Wired AutoConfig is checked.
17.  Click on the security tab and Enable IEEE 802.1X… and for Select and network authentication method, select Microsoft: Protected EAP (PEAP).
18.  Click Properties… and make sure Validate server certificate is checked.  Also check Enable Fast Reconnect and Enable Quarantine checks.  Select Authentication Method should show Secured password (EAP-MSCHAP v2).  Click OK.

Side note: As I was troubleshooting, the NPS log in the expanded Windows 2008 Event Viewer was invaluable to tracking down issues.  You no longer have to read IAS format logs for basic troubleshooting.

Uncategorized | No Comments »

I needed to move a SharePoint 2007 front end from a Windows 2003 32-bit to a Windows 2008 64-bit server while leaving the databases on the existing SQL 2005 server. The contents of this article appear many different places and appear to be the Microsoft accepted method for accomplishing the change: http://technet.microsoft.com/en-us/library/dd622865.aspx. If you want to move everything, that’s the way to go. Since I just wanted to move the SharePoint server itself, and I wanted a way to fall back in case there were compatibility issues, I created a staged approach. It consisted of the following major steps:

1. Build up the new Windows 2008 x64 server, install MOSS 2007 SP1 on it, and create a test site.
2. Install all applications and components that are on the original server onto the new server.
3. Plan downtime and migrate one Site Collection.
4. Test the Site Collection, and then record the exact steps that worked best.
5. Migrate the other Site Collections and decommission the 2003 server.

Since there were many steps and tricks, I wanted to share the full process. These are the assumptions about the SharePoint environment for the purposes of the directions: MOSS 2007; Windows 2003 front-end that also hosts the Central Administration Site; backend SQL 2005 server; needed to do a staged migration to ensure smooth transition for production sites; Maintained same SQL 2005 server on back end, but it would have been the same process with a new server.

1. Document your existing installation. Record such items as:

• third-party web parts
• specialized DLLs – make sure there is a version compiled for 64-bit OS
• templates (stsadm -o enumtemplates)
• packages (stsadm -o enumsolutions)
• presence of static paths
• which web applications are linked to which databases

2. Prepare the Windows 2003 server:
Make sure it is at least upgraded to MOSS SP1. If possible, update it to the latest cumulative update. http://technet.microsoft.com/en-us/library/cc263467.aspx
The Sharepoint installer account will need to be a local administrator on the SQL server, and you will need to log into the SharePoint server as that account during the installation process.
3. Prepare the Windows 2008 x64 sever:
a. Use these instructions to install MOSS 2007 on the new server: http://technet.microsoft.com/en-us/library/cc287748.aspx
b. Add any web parts or other specialized components recorded in Step 1.
c. Configure the permissions.
d. Configure the SSP. It is theoretically possible to migrate an SSP, but I found the procedure to be more trouble than comparing the two side-by-side and replicating the setttings.
4. Perform a test site migration:
a. Make a SQL backup of the content database.
b. Create a blank database with a new database name.
c. Restore the backup into the new database.
d. Create a web application on the new server, and specify the new database name during the creation process.
e. Check the site collection administrators to make sure you are there.
f. If required, do an IIS reset (“iisreset /noforce” at the command line).
g. If using a host header for the site (intranet.company.com), create a DNS entry pointing to the new server with a test site name (intranettest.company.com).
5. After testing of the migration is complete, perform the production migration:
a. Notify users that there will be some downtime.
b. Check that no timer jobs are running.
c. Quiesce the farm for five minutes.
d. Run the preparetomove command for your content database.
e. Make a SQL backup of the content database.
f. Restore the SQL backup over the top of the test database for the new farm.
g. In Central Administration, remove and re-add the content database to the web application.
h. IIS reset.
i. Test internal and external (if applicable) access to the site. Also do some functionality checks: alerts, search (after a full crawl), navigation (static links). Check the Windows event logs for errors.
6. Cleanup:
a. Remove the web application and IIS site from the original farm.
b. Remove the SharePoint installer account from the local administrators on the SQL server.
c. Remove the DNS entry for the testing site.
7. Back up your new environment as soon as it is in a satisfactory state.

One final note: Since I was using a fully qualified domain name for the site name, and I wanted to check functionality of the site from the local server, I ran into the Loopback Check security feature, in which Windows 2008 blocks requests coming from the local machine to prevent reflection attacks. This resulted in a 401 error. As explained here, do not simply set disableloopbackcheck = 1 to get around this. Instead, browse the site from another machine, or use Method 1 from this Microsoft article, in which you specify the host names that should be allowed locally.

Uncategorized | 1 Comment »

Enterprise IT, security | No Comments »