This file will document large changes that anyone tracking development branches
should be aware of. This will also include things like dependency changes or
references to changes in other repositories. Updates are sorted by commit date
descending.

Author: mgoffin
Date:   2014-08-13

    The Sample API has been updated. The Method will default to an empty string
    now to stay in-line with the behavior of the backend handler function. Also,
    parent_md5 is now related_md5 as that option was renamed in the same
    function. Expanded to include related_id and related_type so you can relate
    the Sample to more than just another Sample (common case an Email). Be aware
    if you use this feature to make the appropriate changes.

Author: mgoffin
Date:   2014-08-12

    Updated DEPENDENCIES. We are now supporting Django 1.6.5 (in preparation for
    their 1.7 release), MongoDB 2.6.4, and PyMongo 2.7.2. I also bumped dateutil
    and Yara to 2.1.0 so people using the service push to upgrade and stay on
    the latest code base (new features which will help us enhance the service).

Author: ssnow
Date: 2014-06-23

    Updating CRITs to expand the TAXII service. Within this repository, this
    involves creating a to_stix method for crits objects within
    crits_mongoengine.py. This method will need to be updated as more crits
    objects can be converted into the STIX standard and potentially shared
    via TAXII.

    Related: updating STIX to v1.1.1.0 and CybOX to 2.1.0.5.

Author: mgoffin
Date:   2014-07-16

    The "adduser" management command has been renamed to the "users" management
    command. It's been expanded past just adding users. You can now add and edit
    users. The only thing this doesn't allow you to do is assign sources,
    subscriptions, favorites, or alter user preferences. Please review the new
    list of arguments.

Author: mgoffin
Date: 2014-06-26

    Adding a "Sectors" feature. This will allow you to track which sector(s) a
    top-level object pertains to. The available sectors list is based off of the
    DHS Critical Infrastructure Sectors list and cannot be modified through the
    interface (to allow sharing between instances).

    You will need to run the following to populate the database with the sector
    list:

        python manage.py create_sectors

    There is also a management command to update counts if they get out of whack
    for some reason:

        python manage.py sector_counts

Author: mgoffin
Date: 2014-06-06

    Adding in the start of the Screenshot storage feature. This comes with a
    dependency on python's pillow (v 2.4.0) library. It is also adding an index.
    It is:

    db.screenshots.ensureIndex({'tags': 1}, {'background': true})

    It is possible, but not yet decided, that the GridFS portion of this
    collection should be sharded. Going to think about that a bit while the
    feature matures.

Author: mgoffin
Date:  2014-05.15

    Notifications are getting an overhaul. They are now stored in their own
    separate collection instead of being mixed in with comment and embedded in a
    user's document.

    There is one document per notification, and it contains a list of users that
    notification pertains to.

    The generate_notifications management script has two jobs now:

        - send out email notifications to users who wish to receive them.
        - delete notifications if all of the users it pertains to have been
          notified.

    The create_indexes management command has been updated to include indexes
    for notifications.

        db.notifications.ensureIndex({'users': 1}, {'background': true})
        db.notifications.ensureIndex({'obj_id': 1},
                                     {'background': true,
                                      'expireAfterSeconds': 2592000})

    NOTE: There is an expiration index which is set for 30 days. Notifications
    which have been in the database for 30 days will automatically get deleted
    by MongoDB! If you wish to keep them around indefinitely or want to adjust
    the length of time, please manually generate the indexes or alter your
    create_indexes management script accordingly.

    If you wish to clean up any existing notifications, you'll need to look at
    the following MongoDB commands:

    - User documents which contain notifications:
        db.users.update({}, {$unset: {'notifications': 1}}, {'multi': 1})
        db.users.update({},
                        {$unset: {'unsupported_attrs.notifications': 1}},
                        {'multi': 1})
    - Existing notifications in the comments collection:
        db.comments.remove({'type': 'notification'}, {'multi': 1})

Author: frbapolkosnik
Date:   2014-04-28

    Some LDAP servers might set an urgent flag on some of their messages, and
    cause python-ldap to throw an exception, and not allow an user to log-in.
    This happens with certain Oracle based LDAP servers, and the known solution
    is to import ldap.controls.pwdpolicy.
    
    ldap.controls.pwdpolicy module is not distributed before 2.4.15, and with
    prior version you wil get a message in you crits log:
    "INFO <date> <time> crits.core.user ldap.controls.pwdpolicy not present."

Author: mgoffin
Date:   2014-04-17

    To facilitate the API and to fix some issues, we will now require new
    versions of Django, MongoDB, MongoEngine, and PyMongo. All of the versions
    are available in the DEPENDENCIES file.

    Also, there is an issue with chunk collection indexes. There is a change to
    the Python MongoDB driver which does a sanity check for the chunks index.
    The options for the index differ from the ones we used in create_indexes
    (specifically we added background=true, they did not and require
    unique=true). I have updated create_indexes to support the proper format.
    For anyone with an existing database, you will need to drop and recreate the
    indexes for the objects, sample, and pcaps chunks collections. From mongo
    shell, you can run the following:

    db.objects.chunks.dropIndex({'files_id': 1,'n': 1})
    db.pcaps.chunks.dropIndex({'files_id': 1,'n': 1})
    db.sample.chunks.dropIndex({'files_id': 1,'n': 1})
    db.objects.chunks.ensureIndex( { files_id: 1, n: 1 }, { unique: true } );
    db.pcaps.chunks.ensureIndex( { files_id: 1, n: 1 }, { unique: true } );
    db.sample.chunks.ensureIndex( { files_id: 1, n: 1 }, { unique: true } );

Author: mgoffin
Date: 2014-04-04

    In order to facilitate API Key authorization when using Apache with
    mod_wsgi, you will need to add "WSGIPassAuthorization On" to your Apache
    configuration.

Author: mgoffin
Date:   2014-04-04

    It has come to our attention that there has been some issues with memory
    consumption and lots of swapping since the upgrade to 3.0. This is a result
    of a default change in MongoEngine which uses a cached QuerySet. By default
    it will store the results of each query in memory so multiple iterations of
    the results will not hammer the database. This wound up improving
    performance but at the cost of more significant memory requirements.

    The community has given MongoEngine feedback and in a future version they
    plan on making the default a non-caching QuerySet, but provide the ability
    to cache if you wish to.

    Until such a time comes and we can properly optimize our queries between
    those two options, we have decided to expose a Control Panel option which
    will determine whether or not we should be caching all queries globally.
    By default this option is off so there will be no caching. This means you
    might see slightly slower response times but you'll improve memory/swap
    issues. If you have the hardware and would prefer the performance
    improvements of the query caching, you can enable this in the Control Panel
    and restart your web server.

Author: wxs
Date:   2014-02-03

    The SECRET_KEY setting is no longer stored in the config. It is now
    required that you set that in crits/config/database.py. DO NOT EDIT
    crits/config/database_example.py to make this change. Instead, please
    make copy database_example.py to database.py and put your secret key
    there.

Author: wxs
Date:   2014-01-27

    You need to migrate your samples.

    The 'hashes' sub-document has been removed and individual hashes are
    now top-level objects. Please see the "Sample Migration" section in
    release_notes/3.0.txt for details information.

Author: mgoffin
Date:   2013-01-22

    There has been a change to the create_indexes management script.
    Historically we made a lot of indexes "sparse". These indexes only included
    documents which has a value defined in the indexed field. The issue with
    this now is that our search and sorting capabilities have been enhanced. We
    are trying to search and sort and are coming up with subsets of results due
    to sorting on fields which are sparse indexed. This removes a lot of
    relevent documents from the results.

    The change to create_indexes removes all sparse indexes from being created.
    This will make the index size requirements a little larger, but not anything
    current servers shouldn't be able to handle. If you already have indexes
    made and try to run this script, MongoDB will ignore the creation because an
    index on that field already exists. You will need to manually drop the
    sparse indexes and then use this script to recreate them.

    This is not a requirement for CRITs to continue functioning, but if you
    happen to notice frequent issues with sorting removing some search results,
    this will be your fix.


Author: inray
Date: 2013-01-22

    The XFrameOptionsMiddleware has been added which is used by Django to help
    prevent clickjacking attacks.  This adds the X-Frame-Options HTTP header
    to all outgoing HTTP Responses.  The header restricts the page from being
    loaded within an IFRAME.  Default policy is SAMEORIGIN which allows viewing
    the Control Panel pages.  Deployments where CRITs is loaded inside an
    IFRAME may use the ALLOW-FROM <URI> policy to allow loading CRITs within
    a frame as long as the top-level browsing context is the same as the
    supplied URI.

    See these articles for details:
    http://blogs.msdn.com/b/ieinternals/archive/2010/03/30/combating-clickjacking-with-x-frame-options.aspx
    https://developer.mozilla.org/en-US/docs/HTTP/X-Frame-Options

Author: mgoffin
Date:   2013-12-12

    There are two new dependencies if you want to generate a QR Code on the
    login page for users who are settings up TOTP. You can install them with:

        sudo pip install qrcode
        sudo pip install pillow

    If you do not install these dependencies, users will still get their string
    to setup their authenticator, but will have to manually enter it.

    Also, the Nav menu on the login screen has been removed. Since it only
    contained a link to the password reset page, I included that link near the
    login form on the page instead.

Author: bdray
Date:   2013-12-10

    An effort that started out mainly to extract the dialogs from the core base
    templates led to a fairly lengthy overhaul of many of the dialogs and
    supporting javascript behind them (#921).  The initial intent was to reduce
    the size of the base template given that in any given "load" of a page
    view, you are likely to only use a few of the dialogs, so now most dialogs
    are loaded on demand when you request them.  Future work will add some
    caching in to make this even more performant, but this effort brings us
    closer to that and streamlined a lot of JS.

    For developers, dialogs are currently structured in "mini templates"
    located in the templates/dialogs/ folder.  These dialogs are served by the
    get_dialog function.  Client side, two new JS modules have been introduced
    to make working with them easier.  The first of these modules handles the
    "lazy loading", jquery.ui.dialog-lazy.js -- this module allows you to
    define your dialog mostly as normal, but an added 'href' option is passed
    to the the Dialog function which references the mentioned view.  When a
    user clicks to open the dialog it is ajax fetched and displayed.

    A second module, jquery.ui.dialog-persona adds a built-in option
    flexibility quality called "personas" onto the dialog constructs, enabling
    the easy ability for the same dialog template/form to be reused in multiple
    ways.  For example the 'new' persona can be used to add new entries, and
    the 'update' persona can be used to make changes to existing ones.
    Comments added a 'reply' persona to handle that use case.  By simply
    defining these personas on the dialog and where the dialogClick happens,
    different titles, button labels, setup callback and submit actions can be
    changed to permit this "multi-personality" flexibility.

    As hinted above, opening dialogs is also standardized across the board.
    There are two new classes, 'dialogClick' and 'deleteClick' that were added
    to open up any dialog of the respective type.  So a lot of "on click binding"
    can be alleviated by simply applying the right class.  These functions look for
    additional params on the clicked element to understand the persona and
    dialog they are to open.

    Much of the support code for dialogs has been moved into the dialogs.js
    file, or the respective "apps" file if it is not globally required.  Where
    possible, the functionality has been refactored to support multiple types
    of dialogs with the same backend functions.  Additional work could be done
    to refactor this code in the future.  For example, a lot of code in the
    global namespace could be localized and "contained" to permit both clean
    interfaces as well as possibly enable focused JS unit testing in the future.

    Establishing what dialogs are available also follows a new standardized
    pattern.  You can see this pattern in dialogs.js where stdDialogs,
    fileDialogs, commentsDialogs etc are all defined.  They include the special
    parameters relative to the given dialog, but are essentially all built from
    the stdDialog and stdPersona functions that provide the base functionality
    for all dialogs while allowing you to overlay "specific" options easily.
    If dialogs are specific to certain apps or views, we can save some
    initialization code by only including those in the respective apps JS file
    where "localDialogs" is defined.


Author: mgoffin
Date:   2013-11-19

    With the changes to the directory structure to be more compliant with newer
    versions of Django, this allows us to simplify and use a more default
    django.wsgi file. I have removed the extra crud from that file, but it
    results in the requirement to add the WSGIPythonPath to the Apache config
    files. I have made those changes in our example files, but be aware that you
    will need to add the following line to your httpd.conf (Unbuntu) or your
    ssl.conf file (RHEL):

        WSGIPythonPath /data/crits

    Of course, adjust that line according to your installation if you've
    customized anything.

Author: inray
Date:   2013-10-22

    The directory structure has been updated to stay compatible with newer
    version of Django (1.4+).  The change involved moving most CRITs code to a 
    subdirectory called 'crits' and pointing the configuration and management
    files to the new directory.  The 'DocumentRoot' configuration option is
    the only change required to be made on a normal CRITs installation and is
    located in the Apache configuration files.  You must ensure the Apache
    configuration is updated with the DocumentRoot directive pointing to the
    new path: '/data/crits/crits/extras/www'.

Author: mgoffin
Date:   2013-10-10

    Scripts executed using the `runscript` management command will now require
    authentication against CRITs in order to run. That means you will need a
    valid CRITs login to run scripts. This provides some benefits like auditing
    activity and limiting resulting content based on ACLs (of course this
    doesn't stop people from altering the scripts to bypass this if they have
    access to do so).

    There are several ways to authenticate with `runscript`. One way is doing
    nothing and it will prompt you for a username and password (neither one will
    show the characters as you type). Another way is to use the -u (username)
    and -p (password) options. If you omit one of those it will prompt for the
    missing one. The final way is to use the -e (environ-auth) option. This will
    look for a CRITS_USER and CRITS_PASSWORD environment variable and use those
    to authenticate. Again, if one or more is missing, it will prompt you to
    provide that information. The -e option overrides using the -u or -p option.

    Hopefully this combination of authentication options will provide enough
    ways for people to continue using these scripts in automated tasks. If you
    are currently using these scripts in an automated fashion, note that you
    will need to alter how you execute them. Also if you have any custom scripts
    that leverage runscript, note that you will need to adjust your scripts
    `__init__` method to take a 'username' argument.

Commit: 3d8cc17923fd8dbfd583c67cda782e97bf6bbec4
Author: mgoffin
Date:   2013-09-25

    Certificates have been added as a top-level object. They act very similar to
    PCAPs where there's no metadata extracted from the contents, but they are
    Services-enabled so people can write their own tools and start
    searching/pivoting off the metadata they create.

Commit: fa263d493905a05912102d5e1fff9e627e6a7982
Author: mgoffin
Date:   2013-09-25

    There is a new Control Panel option called "Secure Cookie". It's a Boolean
    value which sets a template variable "secure_cookie" to "True" or "False".
    This can be used by developers to determine if they should be using a cookie
    in a secure or insecure manner. This defaults to "True".

    If you are using the clipboard feature and it seems to not be working, you
    might want to check to ensure that you've set this value properly.

Commit: 979fb5127a50aad26c875518758e3f8404bf3f19
Author: mgoffin
Date:   2013-09-17

    I created an AuditLog class. This class provides the standard structure for
    existing documents in the audit_log collection, and should be used in the
    future for any audit logging. The dates in the database for current audit
    logs were stored as strings. I removed the old prep contents and added a
    migration which converts them all to ISODate. You will need to run the
    following to migrate the dates:

        pythong manage.py prep

    Currently the only thing we read audit_log documents for is determining the
    recent samples on the profile page.

Author: mgoffin
Date:   2013-08-21

    Django 1.5 introduced an ALLOWED_HOSTS list in settings.py. This is required
    to be populated if you set DEBUG to False. Since we did not include this
    already, anyone who sets DEBUG to False and restarts their web server will
    get 500 errors. We have added it and set the default to ['*'] which
    basically makes DEBUG a worthless setting. This is intentional so by default
    anyone can get CRITs running (we can't possibly predict the hostnames of the
    servers people run CRITs on or the URLs they will use to contact it, so this
    seems like a sane default).

    However, the installation process will require that you use setconfig to set
    ALLOWED_HOSTS to a valid value in the database which will override the
    default. Also, you can set this via the UI if you plan on changing DEBUG to
    False.

    When you update your code, please run the following command:

        python manage.py setconfig allowed_hosts "foo"

    Where "foo" is the host/domain name or list of names that your site will
    serve. For more information on this, please visit the following URL:

    https://docs.djangoproject.com/en/1.5/ref/settings/#std%3asetting-ALLOWED_HOSTS

Commit: 827506c3f1e04e7d3f250ec61292baa9c325b372 (crits_dependencies)
Author: wxs
Date:   2013-08-02

    The dependencies for supporting standards have been updated. The new
    versions should improve peroformance when handling large XML files.

    Updated versions are:

        cybox-2.0.0b6
        stix-1.0.0a7
        libtaxii-1.0.105

Commit: 8a11893d0aaaf5a1c111e946dfca2f654099b35b
Author: wxs
Date:   2013-07-29

    We now have beta support for using Amazon S3 support. To enable this please
    read the changes to config/database_example.py (look for "S3") and
    make the corresponding changes in config/database.py.

    If you choose to use S3 please note that it does require the boto package
    be installed, which is available at https://github.com/boto/boto.

Author: mgoffin
Date:   2013-07-24

    There is no longer a requirement on a secondary database for authentication
    and session management. All of this is being done in MongoDB. Before moving
    to this code, however, you need to get and run the migrate_users management
    script. This needs to be done with your secondary database and
    settings/custom_settings in-tact. After the user have been migrated into
    MongoDB you can then update to this code.

        python manage.py migrate_users

    The only config file now to worry about is `config/database.py` which only
    contains the MongoDB connection information.

Author: mgoffin
Date:   2013-07-23

    Updated dependency requirements to Django 1.5 and MongoEngine 0.8.3. The
    modifications necessary for this aren't 100% complete, so these haven't been
    added to the crits_dependencies repository. I also haven't updated the
    services which the necessary changes to their forms.py files for using in
    Django 1.5.

Commit: b2c9846bca48248c040129893728639bc105ffd8
Author: mgoffin
Date:   2013-07-16

    Event Types are now driven by standards. We have removed the ability to add
    new ones from the interface. You will need to run the following to get the
    new Event Types into your db:

        python manage.py create_event_types

    Also, if you have Events in your system which have the old Event Types that
    came with CRITs, the prep management command has been modified to migrate
    them to the appropriate standards version. You can rerun the prep script to
    get these migrated, or do so by hand if you have only a few to take care of.

Commit: feb27c4b34721b705802bfe7459846f5e736c6fa
Author: mgoffin
Date:   2013-07-16

    The 'scripts' directory has been removed from core and added to the
    crits_services repository under 'crits_scripts'. If you were using the
    mapreduces or notifications cronjob scripts, you will need to change them to
    run like this:

        python manage.py mapreduces
        python manage.py generate_notifications

    The 'mongo' directory which was in 'scripts' has been moved to the 'contrib'
    directory.
