ESM package

Entity State Machine, or ESM for short, can be used to process a chain of events. Unlike normal probes that are executed synchronously with the hook, ESM probes are executed asynchronously on the events generated by other probes. This lets you build logic that tracks multiple related events. For example, you can wait for a certain process to write a file and then for another process to read the same file. ESMs can provide powerful detection capabilities but are unable to block operations as they happen as they’re asynchronous. ESMs can only undo or stop operations. For example, you can kill a process in ESM but normal hook probes can stop the process from being created in the first place.

Entity State Machine Probes (ESM) define a set of States, Events and State Transitions triggered by these events. Each ESM is independent of other ESMs with its own set of states and transitions that can be triggered independently of each other even as they may be dependent on the same type of events.

Each ESM starts out in its initial state with the name "initial" being reserved. The initial state isn’t tracked or stored anywhere. It is merely a starting point from which new ESM instances are spawned when the initial state is triggered.

Each ESM state has 2 required attributes: name and triggers. There is also an optional Id attribute that can be set on state creation/transition and can be retrieved via state:getId() method.

triggers attribute is a list of trigger objects each having 3 attributes: event, keyFn and action. event attribute contains the name(s) of the event(s) that triggers the action. keyFn contains a function that, given an event, computes the matching state id that should be triggered by this event. For example, when processing FileOpenEvent, one may choose to use the event.fileHandle attribute as the state id. When processing FileWriteEvent later on, the keyFn that returns event.fileHandle would allow us to match the state object created when processing FileOpenEvent and analyze these two events as part of one pattern.

Since the initial state is not stored anywhere, it doesn’t have to be matched against entities referenced by the triggering events and for this reason keyFn is optional in the initial state triggers. One may still choose to specify it though and when available it will do the opposite, i.e. it will prevent initial state triggers from firing when their keyFn evaluates to match an existing ESM state, i.e. if the same entity has already triggered the initial state earlier.

Finally the action function contains the actions to be triggered when the matching state is found for a given event. Users can put arbitrary logic inside of the action function including additional computations to determine if a state transition is necessary and which state to transition to if so. To transition the ESM to a new state, state:transition(id, stateData, stateName) or state:spawn() methods can be used. When invoked the old ESM state is no longer tracked, while the new one is created using ID, data and stateName passed to it. state:spawn() method is to be used to create a new copy of the ESM instance while state:transition changes the current state of the existing ESM. Since the initial state doesn’t yet have an ESM instance, both spawn and transition methods result in identical behavior. state:spawn() is meant for cases when in the process of processing events we discover new entities that we need to track with each requiring its own ESM state. For example, when a suspect process creates a new process, we need to keep track of both of them. Similarly when a suspicious file is copied, etc.

One can call state:finalize() method to terminate an ESM instance.

Pseudocode

The entire process can be summarized with the following pseudocode.

def process_event_for_esm(event):
  for trigger in esm.state.triggers:
    if event == trigger.event and trigger.keyFn(event) == esm.state_key:
      trigger.action()
      
  for esm in esms:
    for initial_trigger in esm.initial.triggers:
      if event == initial_trigger.event:
        initial_trigger.action()

def state_transition(next_key, next_state):
  esm.state = next_state
  esm.state_key = next_key

Example

In the example below we define a single trigger for the "initial" state. The trigger will fire when FileDownloadEvent is encountered. The action provided for this trigger will simply transition the ESM to the FileDownloadState using event.file.eid as the state ID and event.file entity as the state data. Note that one can provide arbitrary data passed between states depending on the processing needs of the given ESM. The data passed around will be retained for the lifetime of the new state while the event and the previous state’s data will be discarded. One gets access to the data saved in the ESM state using state:getData() method. Note, that while Lua would not prevent you from updating the state or event object being passed into the action function, it is assumed that you would treat them as immutable objects and would create a new state data object with all the needed attributes when transitioning to a new state if you need to modify anything you use.

Esm {
  name = "ProcessCreateFromDownloadedFileEsm",
  probes = {
    {
      name = "FileMonitorProbe"
    },
    {
      name = "ProcessCreateProbe"
    }
  },
  states = {
    {
      name = "initial",
      triggers = {
        {
          event = "FileDownloadEvent",
          -- initial state doesn't need keyFn.  It applies to any matching event regardless of its key value
          action = function(state, event)
            -- Transition to the FileDownloadState with the given id iff it doesn't yet exist
            state:transition(event.file.eid, "FileDownloadState")
          end
        }
      }
    },
    {
      name = "FileDownloadState",
      triggers = {
        {
          event = "FileMoveEvent",
          keyFn = function(event)
            return event.file.eid
          end,
          action = function(state, event)
            -- Transition the current state object to the same state with new ID and data
            state:transition(event.dstFile.eid, "FileDownloadState")
          end
        },
        {
          event = "FileCopyEvent",
          -- When provided keyFn will be used to compute the event key and 
          -- would only trigger actions on those states that match it
          keyFn = function(event)
            return event.file.eid
          end,
          action = function(state, event)
            -- File created by browser was copied. 
            -- state:spawn() is just like transition(), but will create
            -- a copy of the ESM state with the original one still being available.
            state:spawn(event.dstFile.eid, "FileDownloadState")
          end
        },
        {
          event = "FileDeleteEvent",
          keyFn = function(event)
            return event.file.eid
          end,
          action = function(state, event)
            -- File created by browser was deleted. Forget about it.
            state:finalize()
          end
        },
        {
          event = "ProcessCreateEvent",
          keyFn = function(event)
            return event.process.backingFile.eid
          end,
          action = function(state, event)
            -- A process was created from the file entity. Consider such processes to be potentially suspicious.
            Alert(
               "ProcessCreateFromDownloadedFileEvent",
               {
                  actorProcess = event.actorProcess,
                  process = event.process
               }
            ):send(EventChannel.splunk, EventChannel.file)
          end
        }
      }
    }
  }
}