Organizations may not have to work hard to initially develop this trust relationship with their customer.  What an organization may not be doing is working hard to maintain this trust relationship.  If the integrity of these assets are challenged, there is a potential risk of destroying this trust relationship, which could be cause for a costly recovery attempt of this customer trust, or complete customer loss.  Ensuring the integrity of your organization’s digital assets should be considered a foundational component of any organization’s security practice.

When we speak of digital assets, we not only include the traditional media data types including images, audio, and video, but all other content produced by your company that exists in a persistent state on disk.  In a web server environment we would most likely be dealing the with following types of data:

• Html, Javascript, CSS, and embedded web objects
• PHP, Perl, .NET, and other application code
• Non-HTML Document types such as PDF or Office files
• Images, Audio, and Video

We were recently employed to review the state of a customer’s environment following a recent intrusion.  As with a portion of the environments we work in today, the services provided included web content and application delivery provided by Apache running on Linux.  While Apache does provide access logging for files requested from it, it does not maintain state regarding the integrity of the files it serves. Following an intrusion, identifying the the attack vector is extremely important in providing future security of this environment but should not be the only consideration.

The following questions need to be asked concurrently:

• What organizational or customer data was exposed?

• Was any data modified and what is the potential impact to our customers?

In high traffic environments, it can be extremely difficult to answer both of these questions quickly, which in turn can prolong the delivery of customer communication or notification for external entities that may have a stake in the exposed data.  To speed up the time to derive an answer to both of these questions, there are two methods that are available to expedite this process.

First, the majority of server operating systems in production today have kernel facilitated auditing capability bundled with the operating system.  Linux provides the Linux Audit Subsystem.  Microsoft Windows Server including 2003 and 2008 provide auditing capability.  Solaris, MacOS X, and the BSD family of operating systems also implement audit facilities.  Each of these respective implementations provide the ability to monitor file access and modification events and produce audit trail which can be used to quickly determine which critical assets were accessed or modified.  Although the deployment of auditing policy is not trivial, the benefit can easily be measured if file integrity is violated and you are able to effectively determine the targeted assets and the associated scope of exposure.

Secondly, organizations should deploy a file integrity monitoring system such as Tripwire or Samhain.  These systems utilize one way cryptographic functions (also known as message digest algorithms) such as MD5, SHA1, SHA2, or Tiger, to create a catalog of computed hashes of  files covered by the monitoring software’s defined policy.  Following the a baseline definition process, these systems monitor filesystem changes against prior hash calculations, and in some cases, against known bad hash values associated with exploit, rootkit code, and other potential malware.   When a change event is discovered, notifications can be delivered to those accountable.

The data generated by each of these tools should be streamed via encrypted transport to a centralized syslog server.   This centralized server should exist in a logically distinct network segment from all other nodes in the environment.   Because this server essentially becomes the gold copy record for file integrity in your environment as related to file assets, extreme care should be used to ensure the validity of logs captured.  This includes utilizing file integrity monitoring and limiting access to those who have a need to know.

The employment of these systems do not provide protection against intrusion but can ease the burden of cleaning up the mess and help organizations identify impact.  While this does not completely mitigate potential loss of trust with your customers, it allows you to effectively measure whether or not their trust was violated and the overall level of exposure.  In the future, we will be providing direction on how these logs can be utilized to provide real-time alerting of an attack in progress, and what you can do to decrease your time to react.

This is how to change the categories:
1. Go to the calendar web part from the browser.
2. Under the Calendar Tools tab, click the Calendar tab.
3. Choose List Settings.
4. Scroll down to the Columns heading.
5. Click Category.
6. Under the Additional Column Settings heading, there is a text box with the categories, and above the box reads: “Type each choice on a separate line.”
7. Add or remove categories, then click OK.

I’ve been working with a customer who’s exceeded the throughput capabilities of their Equalizer E350si load balancer.   Although the marketing materials will tell you that this unit is capable of throughput up to 700Mbps (hah!),  we were maxing out and dropping packets above 50 Mbps.

Rant:  You see, the problem lies in Coyote’s system architecture.  This E350si is powered by a NetBurst-architecture Pentium 4 2.8Ghz, with HyperThreading disabled.   Coyote uses a FreeBSD-4 based kernel, which was well known for it’s slow timers, slow interrupt handling, and immature device polling implementation.  In this classic system architecture, each incoming packet generates an Interrupt Request (IRQ), which must be serviced by the CPU in a time-slotted fashion.   So what we have is a load balancer which reports it’s CPU as being mostly idle,  but in reality cannot handle packets quickly enough.   THIS IS NOT HOW YOU DESIGN NETWORK EQUIPMENT, PEOPLE.   End rant.

Good news:   In version 8, Coyote introduced a new mode of operation called DSR, or Direct Server Return.  DSR is quite clever, really, because it get’s around Coyote’s packet handling limitation (to a big degree) by handling the incoming network packets, but allowing the web servers to respond directly to clients.   This cuts the number of TCP packets the Coyote has to process in half,  and cuts the number of Ethernet frames by much more if you consider that the return packets are much larger.

Here’s how it works.   In a traditional setup,  the Coyote receives a packet on its external interface (em1), makes a load balancing decision, and then forwards to the packet along to a host behind its internal interface (em0).   Most shops NAT here as well, for security and/or IP address conservation reasons.   So the Coyote must perform Layer 2 – 4 (or 7) processing of the packet as it receives it,   then make a load balancing decision,  then translate the packet (that’s the T in NAT),  then re-process the packet going out the internal interface.   Then, rinse, lather, repeat for the return packet.   Such is the life of a typical load balancer.

In DSR mode, you start by chopping off the Internal interface of the load balancer altogether and eliminating NAT.   This requires moving your webservers onto publicly routable IP addresses, so please make sure they are firewalled properly.   Now you have your load balancer and webservers all on the same ethernet segment.   You create a VIP (Virtual IP) on the load balancer, and then add that SAME VIP address as a loopback address to the webservers!

You’re probably scratching your head, wondering how this is going to work.  I know that I was.  Here’s the magical part.   Only the load balancer responds to ARP requests for the VIP.  The webservers have Apache listening on the VIP address,  but don’t respond to ARP requests at all on that address.   Each incoming packet is sent from the router to the MAC address of the load balancer,  which then makes a load balancing decision and then sends an identical copy of that packet to the MAC address of the web server.   Let me say that again.  The load balancer performs no more translation, it literally just copies the packet over to the webserver.   Since the source MAC address is unchanged,  the web server replies directly to the router and skips the load balancer entirely.

Sounds a bit scary, but works well.  Except for one thing.   In their brilliance,  the Coyote folks created a section in the Manual with configuration instructions for “Linux/Unix Systems”, but ACTUALLY put in instructions for BSD-like systems only.  Who runs FreeBSD anymore?   DON’T TRY THESE INSTRUCTIONS ON A LINUX SERVER UNLESS YOU WANT TO LOCK YOURSELF OUT.

On Linux,  the correct way to create the loopback address is by adding a “labelled” loopback interface,  but ALWAYS set the netmask of your new interface to “255.255.255.255″.   If you match the netmask of the VIP,  your webserver will stop responding to packets on it’s external interface.  Very bad.

So,  assume your public VIP address is 12.34.56.78  (fake, to protect the innocent),  and your webserver’s address is 12.34.56.80.   Create a loopback address like so:

/sbin/ifconfig lo:vip inet 12.34.56.78 netmask 255.255.255.255

Then, the output of “ifconfig -a” looks something like this:

eth0      Link encap:Ethernet  HWaddr 00:40:A4:8E:B0:1A
inet addr:12.34.56.80  Bcast:12.34.56.255  Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
RX packets:47131905 errors:0 dropped:0 overruns:0 frame:0
TX packets:77804088 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:5111837334 (4875.0 Mb)  TX bytes:104047655003 (99227.5 Mb)
Interrupt:177

lo        Link encap:Local Loopback
inet addr:127.0.0.1  Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING  MTU:16436  Metric:1
RX packets:1311164 errors:0 dropped:0 overruns:0 frame:0
TX packets:1311164 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:439975560 (419.5 Mb)  TX bytes:439975560 (419.5 Mb)

lo:vip    Link encap:Local Loopback
inet addr:12.34.56.78  Mask:255.255.255.255
UP LOOPBACK RUNNING  MTU:16436  Metric:1

If it all works, you should be able to confirm correct operation by using tcpdump or Wireshark/Ethereal on the webserver, and verifying that the SOURCE address is your VIP address and you’re seeing lots of 200 OK messages.

tshark -n -i eth0 -R http.response port 80

Unfortunately, WordPress isn’t setup for high performance operation out of the box.  Each page request is very CPU and database intensive.

Under benchmarking, we discovered that each of the customer’s HP DL385 servers could serve between 5-10 WordPress page view’s per second, depending on the page.  (For the pedantic,  I’m considering anything that hits PHP as a page view.)  And this is AFTER I had put major effort into MySQL performance tuning.    Something had to be done, 5 page views per second is just not going to cut the mustard.

In comes the WP-Super-Cache plugin to save the day.   WP-Super-Cache is a plugin which seems like it should be installed with every WordPress instance by default.   It writes out entire pages to static .html files,  and then instructs Apache to serve up the static .html files directly (using mod_rewrite),  therefore avoiding any CPU-gobbling calls to PHP or the database.   But WP-Super-Cache is smart,  it automatically expires cached pages when the content is updated  (by the author, or via comments).

As a result,  we went from 5-10 page views per second to between 500-2000 theoretical page views per second.   At this point we were hitting bandwidth bottlenecks,  which is where I like to be.   As long as webservers can serve up enough data to fill their own pipe, you have happy system administrators (and UNhappy network administrators).