Task53Start - Examine owa audit-logs and do a rulefile for owa with firefox

See TasksExplained? for more infos about tasks. This task is based on the template at TaskTemplateStart

Status : open
Category : gui
Date opened : 2007-04-02
Description : see above
Referenced in stories/tasks : WebStatistics Story15Start
Testsuite : none
Commited revision : 208 and above

Remarks :

---

Changelog

2007-04-12 - starting up

Got OWA audit logs for IE and FF from Rick:

We've tested with Internet Explorer 7 so we could use the premium OWA
client with WebDav enabled. We've also tested with FireFox 2 to use the
basic client that does not have WebDav enabled.

We've also tested with Dutch regional settings enabled, so you'll see
Dutch foldernames in the logs, like 'Postvak IN' for 'Inbox' and
'Agenda' for 'Calendar'. Foldernames should not be used in the ruleset
otherwise you end up with several localised OWA-rulesets.

Extracting FF methods and paths

I decide to work on a firefox ruleset for a (simpler) start:

audit-log-parser.rb -l remo.FF > firefox-ruleset/request-list
cat firefox-ruleset/request-list | sed -e "s/^[0-9]//" -e "s/^[0-9]//" -e "s/^[0-9]//" -e "s/^: //" -e "s/?.*//" -e "s/HTTP.*//"
cat method-path-list  | sort | uniq > method-path-list-sorted

->

GET /exchange 
GET /exchange/ 
GET /exchange/xxx/
GET /exchange/xxx//
GET /exchange/xxx//Verwijderde%20items
...

Extracting the regex paths:

• /exchweb/img/[0-9a-zA-Z_-]{1,20}.gif
  0-9 does not appear in the filenames, but this is safer.
• /exchweb/[0-9.]{1,16}/controls/[0-9a-zA-Z_-]{1,20}.(css|js)
  0-9 and capitalas do not appear in the filenames, but this is safer.
• doing the same for the other paths with http method GET

The GET requests as regexes

GET /exchange[/]? 
GET /exchange/[0-9a-zA-Z_-]{1,16}/[0-9a-zA-Z/%_-]{0,120}(.EML|.eml)?/?
GET /exchweb/bin/auth/owalogon.asp
GET /exchweb/[0-9.]{1,16}/controls/[0-9a-zA-Z_-]{1,20}.(css|js)
GET /exchweb/help/[0-9a-zA-Z_-/]{1,30}.(css|gif|htm)
GET /exchweb/[iImMgG]/[0-9a-zA-Z_-]{1,20}.gif 
GET /exchweb/themes/\d/[0-9a-zA-Z_-]{1,20}.gif
GET /favicon.ico 
GET /public[/]?
GET /public/Internet%20Newsgroups/

2007-04-19 - More stuff

The first logfile did not contain POST parameters. More logfiles coming in 20070418/19.

2007-04-20 - Table of paths and parameters (so far firefox only)

GET /exchange[/]?
  Cookies:
    sessionid: [0-9a-z-]{36}(:[0-9x]{5})?   ->   set as custom regex
      The last optional part has been present twice as ":0x809"
    cadata: \"[0-9a-zA-Z+/]{0,61}={0,2}\"   ->   set as custom regex
      Looks like base64, but it is not decodeable. Still it uses the same characters.

GET /exchange/[0-9a-zA-Z-_]{1,16}[0-9a-zA-Z/\x20\x7B\x7D-_]{0,120}(.EML|.eml)?[/]?
  Cookies:
    sessionid: [0-9a-z-]{36}(:[0-9x]{5})?   ->    set as custom regex
      The last optional part has been present twice as ":0x809"
    cadata: \"[0-9a-zA-Z+/]{0,61}={0,2}\"   ->    set as custom regex
      Looks like base64, but it is not decodeable. Still it uses the same characters.
  Querystring parameters:
    [A-Z][A-Z]: [0-9a-zA-Z-_.]{0,32}   ->   set as custom regex
      seems to be an ldap parameter, this is a guess
    [Cc]md: [a-z]{1,18}   ->   Letters/Numbers, max. 32 characters
    Create: \d   ->   Integer, max. 16 characters
    Error: [0-9]{1,5}   ->   Integer, max. 16 characters
    Page: [0-9]{1,3}   ->   Integer, max. 16 characters
    ShowFolders: [0-9]{1,3}   ->   Integer, max. 16 characters
    SortBy: [0-9a-zA-Z-/\x20_]{1,32}   ->   set as custom regex
    SortOrder: (ascending|descending)   ->   Letters/Numbers, max. 16 characters
    Target: [0-9a-zA-Z\x20-_]{0,32}   ->   Letters/Numbers/Space/-/_, max. 32 characters
      This value was empty in my audit file. Hard to tell. Choosing a good default; could be 
    Type: [a-zA-Z]{1,18}   ->   Letters/Numbers, max. 32 characters
    View: [0-9a-zA-Z\x20-_]{0,64}   ->   set as custom regex
    [dmy]: [0-9]{1,2}   ->   Integer, max. 16 characters
    sessid: [0-9a-z-]{36}   ->   set as custom regex
    MailToURI: [0-9a-zA-Z/\x20.]{0,64}   ->   set as custom regex

POST /exchange/[0-9a-zA-Z-_]{1,16}[0-9a-zA-Z/\x20\x7B\x7D-_]{0,120}(.EML|.eml)?[/]?
  Cookies:
    sessionid: [0-9a-z-]{36}(:[0-9x]{5})?   ->   set as custom regex
      The last optional part has been present twice as ":0x809"
    cadata: \"[0-9a-zA-Z+/]{0,61}={0,2}\"   ->   set as custom regex
      Looks like base64, but it is not decodeable. Still it uses the same characters.
  Querystring parameters:
    Cmd: [a-z]{1,18}   ->   Letters/Numbers, max. 32 characters
    Embedded: \d   ->   Integer, max. 16 characters
    Target: [0-9a-zA-Z\x20-_]{0,32}   ->   Letters/Numbers/Space/-/_, max. 32 characters
      This value was empty in my audit file. Hard to tell. Choosing a good default.
  Post parameters:
    ActWork_(Unit|Value): \d   ->   Integer, max. 16 characters
    Action: [a-zA-Z]{1,16}   ->   Letters/Numbers, max. 16 characters
    AttachName: \d   ->   Integer, max. 16 characters
    Cmd: [a-z]{1,18}   ->   Letters/Numbers, max. 32 characters
      This can not be combined with the other Cmd* parameters, as this is being checked against the
      query string parameter Cmd (crosscheck)
    [Cc]md[a-zA-Z]{1,16}: [a-z]{1,18}   ->   Letters/Numbers, max. 32 characters
    DATACHANGED: \d   ->   Integer, max. 16 characters
    Email\d: [0-9a-zA-Z\x40-_.]{0,64}   ->   set as custom regex
    Embedded: \d   ->   Integer, max. 16 characters
    EstWork_(Unit|Value): \d   ->   Integer, max. 16 characters
    FORMTYPE: [a-z]{1,18}   ->   Letters/Numbers, max. 32 characters
    FldID: [0-9a-zA-Z-\x25\x2F_.]{1,32}   ->   set as custom regex
    Folder(Name|Type): [0-9a-zA-Z-\x2F_.]{1,32}   ->   set as custom regex
    FormType: [a-zA-Z]{1,16}   ->   Letters/Numbers, max. 16 characters
    Importance: \d   ->   Integer, max. 16 characters
    Index: [0-9a-zA-Z\x20-_]{0,32}   ->   Letters/Numbers/Space/-/_, max. 32 characters
      This value was empty in my audit file. Hard to tell. Choosing a good default; could be 
    Link_[a-zA-Z]{1,16}: https?[0-9a-zA-Z-\x26\x3A\x2F\x3D\x3F_.]{1,128}   ->   set as custom regex
    Msg(Bcc|Cc|To|ID|Class): [0-9a-zA-Z-\x25\x2F\x7B\x7D\x40_.]{0,64}   ->   set as custom regex
    Oof(Reply|State): [0-9a-zA-Z-\x20_]{0,32}   ->   Letters/Numbers/Space/-/_, max. 32 characters
      The value OofReply was empty in my audit file. Hard to tell. Choosing a good default
    Optional: [0-9a-zA-Z-\x20_]{0,32}   ->   Letters/Numbers/Space/-/_, max. 32 characters
    Page: \d   ->   Integer, max. 16 characters
    ParentFolder: https?[0-9a-zA-Z-\x25\x26\x3A\x2F\x3D\x3F_.]{1,128}   ->   set as custom regex
    ReadForm: \d   ->   Integer, max. 16 characters
    ReceiptOptions: \d   ->   Integer, max. 16 characters
    Required: [0-9a-zA-Z-\x40_.]{0,64}   ->   set as custom regex
    Resource: [0-9a-zA-Z-\x20_]{0,32}   ->   Letters/Numbers/Space/-/_, max. 32 characters
    SENDUPDATE: \d   ->   Integer, max. 16 characters
    SubmitCreds: Log\x20On    ->   Letters/Numbers/Space/-/_, max. 16 characters
      This is "Log+On" thus "Log On", thus "Log\x20On"
    ViewName: [0-9a-zA-Z-\x25_]{0,32}   ->   set as custom regex
    WebPage: [0-9a-zA-Z-\x20_]{0,32}   ->   Letters/Numbers/Space/-/_, max. 32 characters
    _chkEnableJunkFiltering: [a-zA-Z]{0,16}   ->   Letters/Numbers, max. 16 characters
    (addto|delfrom)(js|tr|ts): [0-9a-zA-Z-\x3B\x40_.]{0,64}   ->   set as custom regex
    advancerecur: [a-zA-Z]{0,16}   ->   Letters/Numbers, max. 16 characters
    chk(BlockExtern|TrustedContacts): [a-zA-Z]{0,16}   ->   Letters/Numbers, max. 16 characters
    completed_[a-zA-Z]{1,8}: \d{1,4}   ->   Integer, max. 16 characters
    contactFieldBaseIndex: [0-9a-zA-Z-_]{0,32}   ->   Letters/Numbers/Space/-/_, max. 32 characters
    daily_type: [a-zA-Z]{0,16}   ->   Letters/Numbers, max. 16 characters
    days: \d   ->   Integer, max. 16 characters
    destination: https?[0-9a-zA-Z-\x3A\x2F_.]{1,128}   ->   set as custom regex
    dlsignaturemodified: \d   ->   Integer, max. 16 characters
    dt[a-zA-Z_]{1,32}: [\d\x3A]{1,16}   ->   set as custom regex
    enable(task)?reminders: [a-z]{1,16}   ->   Letters/Numbers, max. 16 characters
    exception: [a-zA-Z]{0,16}   ->   Letters/Numbers, max. 16 characters
    flags: \d   ->   Integer, max. 16 characters
    formrecur_type: \d    ->   Integer, max. 16 characters
    http\x3A\x2F\x2Fschemas.microsoft.com\x2F[0-9a-zA-Z\x2F]{1,64}: [0-9a-zA-Z-\x20\x3A\x40\x2F_.]{0,64}   ->   set as custom regex
    includecontacts: \d   ->   Integer, max. 16 characters
    junkemailstate: \d   ->   Integer, max. 16 characters
    (month|recur|reminder)[0-9a-zA-Z_]{1,16}: [0-9a-z\x3A]{1,16}   ->   set as custom regex
    monthly_type: [a-z]{0,16}   ->   Letters/Numbers, max. 16 characters
    password: [.]{16}   ->   Anything, max. 16 characters
    pctc: \d   ->   Integer, max. 16 characters
    radio_[a-z]{1,16}: [0-9a-z]{1,16}   ->   Letters/Numbers, max. 16 characters
    responserequested: [a-z]{1,16}   ->   Letters/Numbers, max. 16 characters
    safelistsonly: \d    ->   Integer, max. 16 characters
    sel[a-zA-Z]{1,32}: [0-9a-zA-Z-\x20\xA0_]{0,32}   ->   set as custom regex
    skipoccur: [a-zA-Z]{0,16}   ->   Letters/Numbers, max. 16 characters
      This value was empty in my audit file. Hard to tell. Choosing a good default
    sliding_[a-z]{1,16}: \d   ->   Integer, max. 16 characters
    task[a-z_]{1,16}: \d   ->   Integer, max. 16 characters
    trusted: \d   ->   Integer, max. 16 characters
    txt[a-zA-Z]{1,16}: [0-9a-zA-Z-\x40_.]{0,64}   ->   set as custom regex
    update(js|tr|ts): [a-zA-Z]{0,16}   ->   Letters/Numbers, max. 16 characters
      This value was empty in my audit file. Hard to tell. Choosing a good default
    urn[0-9a-zA-Z\x3A]{1,64}: [0-9a-zA-Z-\x0D\x0A\x20\x3A_.]{1,64}   ->   set as custom regex
    username: [0-9a-zA-Z-_]{1,16}   ->   Letters/Numbers/Space/-/_, max. 16 characters
    week(ly|s)(_type)?: [0-9a-zA-Z-_]{1,16}   ->   set as custom regex
    year[0-9a-z_]{0,10}: [0-9a-zA-Z-_]{1,16}   ->   set as custom regex

GET /exchweb/[0-9.]{1,16}/controls/[0-9a-zA-Z-_]{1,20}.(css|js) 
  Cookies:
    sessionid: [0-9a-z-]{36}(:[0-9x]{5})?   ->   set as custom regex
      The last optional part has been present twice as ":0x809"
    cadata: \"[0-9a-zA-Z+/]{0,61}={0,2}\"   ->   set as custom regex
      Looks like base64, but it is not decodeable. Still it uses the same characters.

POST /exchweb/bin/auth/owaauth.dll 
  Cookies:
    sessionid: [0-9a-z-]{36}(:[0-9x]{5})?   ->   set as custom regex
      The last optional part has been present twice as ":0x809"
    cadata: \"[0-9a-zA-Z+/]{0,61}={0,2}\"   ->   set as custom regex
      Looks like base64, but it is not decodeable. Still it uses the same characters.
  Post parameters:
    SubmitCreds: Log\x20On    ->   Letters/Numbers/Space/-/_, max. 16 characters
      This is "Log+On" thus "Log On", thus "Log\x20On"
    destination: https?[0-9a-zA-Z-\x3A\x2F_.]{1,128}   ->   set as custom regex
    flags: \d   ->   Integer, max. 16 characters
    password: .{0,16}   ->   Anything, max. 16 characters
    trusted: \d   ->   Integer, max. 16 characters
    username: [0-9a-zA-Z-_]{1,16}   ->   Letters/Numbers/Space/-/_, max. 16 characters

GET /exchweb/bin/auth/owalogon.asp 
  Cookies:
    sessionid: [0-9a-z-]{36}(:[0-9x]{5})?   ->   set as custom regex
      The last optional part has been present twice as ":0x809"
    cadata: \"[0-9a-zA-Z+/]{0,61}={0,2}\"   ->   set as custom regex
      Looks like base64, but it is not decodeable. Still it uses the same characters.
  Querystring parameters:
    reason: [0-9]{1}   ->   Integer, max. 16 characters
    url: [0-9a-zA-Z{}/\x20-_:.]{0,120}   ->   set as custom regex

GET /exchweb/help/[0-9a-zA-Z-_/]{1,30}.(css|gif|htm) 
  Cookies:
    sessionid: [0-9a-z-]{36}(:[0-9x]{5})?   ->   set as custom regex
      The last optional part has been present twice as ":0x809"
    cadata: \"[0-9a-zA-Z+/]{0,61}={0,2}\"   ->   set as custom regex
      Looks like base64, but it is not decodeable. Still it uses the same characters.

GET /exchweb/[iI][mM][gG]/[0-9a-zA-Z-_]{1,20}.gif 
  Cookies:
    sessionid: [0-9a-z-]{36}(:[0-9x]{5})?   ->   set as custom regex
      The last optional part has been present twice as ":0x809"
    cadata: \"[0-9a-zA-Z+/]{0,61}={0,2}\"   ->   set as custom regex
      Looks like base64, but it is not decodeable. Still it uses the same characters.

GET /exchweb/themes/\d/[0-9a-zA-Z-_]{1,20}.gif 
  Cookies:
    sessionid: [0-9a-z-]{36}(:[0-9x]{5})?   ->   set as custom regex
      The last optional part has been present twice as ":0x809"
    cadata: \"[0-9a-zA-Z+/]{0,61}={0,2}\"   ->   set as custom regex
      Looks like base64, but it is not decodeable. Still it uses the same characters.

GET /favicon.ico 
  Cookies:
    sessionid: [0-9a-z-]{36}(:[0-9x]{5})?   ->   set as custom regex
      The last optional part has been present twice as ":0x809"
    cadata: \"[0-9a-zA-Z+/]{0,61}={0,2}\"   ->   set as custom regex
      Looks like base64, but it is not decodeable. Still it uses the same characters.

GET /public/Internet\x20Newsgroups/ 
  Cookies:
    sessionid: [0-9a-z-]{36}(:[0-9x]{5})?   ->   set as custom regex
      The last optional part has been present twice as ":0x809"
    cadata: \"[0-9a-zA-Z+/]{0,61}={0,2}\"   ->   set as custom regex
      Looks like base64, but it is not decodeable. Still it uses the same characters.
  Querystring parameters:
    Cmd: [a-z]{1,18}   ->   Letters/Numbers, max. 32 characters

GET /public[/]?  
  Cookies:
    sessionid: [0-9a-z-]{36}(:[0-9x]{5})?   ->   set as custom regex
      The last optional part has been present twice as ":0x809"
    cadata: \"[0-9a-zA-Z+/]{0,61}={0,2}\"   ->   set as custom regex
      Looks like base64, but it is not decodeable. Still it uses the same characters.
  Querystring parameters:
    Cmd: [a-z]{1,16}   ->   Letters/Numbers, max. 16 characters
    Error: [0-9]{1,5}   ->   Integer, max. 16 characters
    Page: [0-9]{1,3}   ->   Integer, max. 16 characters
    Ref: [0-9a-zA-Z{}/\x20-_:.]{0,120}   ->   set as custom regex
    SortBy: [0-9a-zA-Z-/\x20_]{1,32}   ->   set as custom regex
    SortOrder: (ascending|descending)   ->   Letters/Numbers, max. 16 characters
    View: [0-9a-zA-Z]{1,32}   ->   Letters/Numbers, max. 32 characters
    mbxguid: [0-9a-zA-Z-]{36}   ->   set as custom regex
    start: [0-9a-zA-Z-\x20:]{0,25}   ->   set as custom regex
    u: [0-9a-zA-Z@-_.]{1,64};[0-9]{0,3}   ->   set as custom regex

POST /public/ 
  Cookies:
    sessionid: [0-9a-z-]{36}(:[0-9x]{5})?   ->   set as custom regex
      The last optional part has been present twice as ":0x809"
    cadata: \"[0-9a-zA-Z+/]{0,61}={0,2}\"   ->   set as custom regex
      Looks like base64, but it is not decodeable. Still it uses the same characters.
  Post parameters:
    Action: [a-zA-Z]{1,16}   ->   Letters/Numbers, max. 16 characters
    Cmd: [a-z]{1,18}   ->   Letters/Numbers, max. 32 characters
    Folder(Name|Type): [0-9a-zA-Z-\x2F_.]{1,32}   ->   Letters/Numbers, max. 16 characters