At this point in time, Your organization most likely uses its website to deliver key business data to your customers. This could include the delivery of product marketing information, contact information, or product support documentation. Your product may be your website if you deliver your application in a SaaS or cloud based distribution model. As a customer, when I pay for something, I am telling that company that I trust they can deliver not only what they sold me as a product, but all necessary services that come with supporting that product. In other words, I am telling that company that I trust them. As part of this trust relationship, we expect that the systems that enable this content delivery, such as web servers, are trustworthy, and the integrity of the content has also been ensured.
Organizations may not have to work hard to initially develop this trust relationship with their customer. What an organization may not be doing is working hard to maintain this trust relationship. If the integrity of these assets are challenged, there is a potential risk of destroying this trust relationship, which could be cause for a costly recovery attempt of this customer trust, or complete customer loss. Ensuring the integrity of your organization’s digital assets should be considered a foundational component of any organization’s security practice.
When we speak of digital assets, we not only include the traditional media data types including images, audio, and video, but all other content produced by your company that exists in a persistent state on disk. In a web server environment we would most likely be dealing the with following types of data:
- Html, Javascript, CSS, and embedded web objects
- PHP, Perl, .NET, and other application code
- Non-HTML Document types such as PDF or Office files
- Images, Audio, and Video
We were recently employed to review the state of a customer’s environment following a recent intrusion. As with a portion of the environments we work in today, the services provided included web content and application delivery provided by Apache running on Linux. While Apache does provide access logging for files requested from it, it does not maintain state regarding the integrity of the files it serves. Following an intrusion, identifying the the attack vector is extremely important in providing future security of this environment but should not be the only consideration.
The following questions need to be asked concurrently:
- What organizational or customer data was exposed?
- Was any data modified and what is the potential impact to our customers?
In high traffic environments, it can be extremely difficult to answer both of these questions quickly, which in turn can prolong the delivery of customer communication or notification for external entities that may have a stake in the exposed data. To speed up the time to derive an answer to both of these questions, there are two methods that are available to expedite this process.
First, the majority of server operating systems in production today have kernel facilitated auditing capability bundled with the operating system. Linux provides the Linux Audit Subsystem. Microsoft Windows Server including 2003 and 2008 provide auditing capability. Solaris, MacOS X, and the BSD family of operating systems also implement audit facilities. Each of these respective implementations provide the ability to monitor file access and modification events and produce audit trail which can be used to quickly determine which critical assets were accessed or modified. Although the deployment of auditing policy is not trivial, the benefit can easily be measured if file integrity is violated and you are able to effectively determine the targeted assets and the associated scope of exposure.
Secondly, organizations should deploy a file integrity monitoring system such as Tripwire or Samhain. These systems utilize one way cryptographic functions (also known as message digest algorithms) such as MD5, SHA1, SHA2, or Tiger, to create a catalog of computed hashes of files covered by the monitoring software’s defined policy. Following the a baseline definition process, these systems monitor filesystem changes against prior hash calculations, and in some cases, against known bad hash values associated with exploit, rootkit code, and other potential malware. When a change event is discovered, notifications can be delivered to those accountable.
The data generated by each of these tools should be streamed via encrypted transport to a centralized syslog server. This centralized server should exist in a logically distinct network segment from all other nodes in the environment. Because this server essentially becomes the gold copy record for file integrity in your environment as related to file assets, extreme care should be used to ensure the validity of logs captured. This includes utilizing file integrity monitoring and limiting access to those who have a need to know.
The employment of these systems do not provide protection against intrusion but can ease the burden of cleaning up the mess and help organizations identify impact. While this does not completely mitigate potential loss of trust with your customers, it allows you to effectively measure whether or not their trust was violated and the overall level of exposure. In the future, we will be providing direction on how these logs can be utilized to provide real-time alerting of an attack in progress, and what you can do to decrease your time to react.