gdl
    title     = AMQ RFC 006
    subtitle  = The AMQP/Fast Wire-Level Protocol
    product   = OpenAMQ
    author    = iMatix Corporation <amq@imatix.com>
    date      = 2005/07/05
    copyright = Copyright (c) 2004-2005 JPMorgan
    version   = 0.9a5
end gdl

Cover
*****

State of This Document
======================

This document is a request for comments.  Distribution of this document is
currently limited to use by iMatix, JPMorgan, and approved partners under
cover of the appropriate non-disclosure agreements.

This document is a formal standard.
This document is ready for review.

Recent Change History
---------------------

2005/06/06 - 0.9a5:

- Fixes as we implement ASL layers.
- Removed synctag and synchronisation request functionality (for now).

2005/06/06 - 0.9a4:

- Various corrections to text discussion.

2005/06/06 - 0.9a3:

- Added the Basic content class.

2005/06/06 - 0.9a2:

- Full definition of Exchange and Queue classes and updating of all other
  classes and methods to remove Destination and Subscription concepts.

2005/06/05 - 0.9a1:

- Formulation of "exchange" concept for matching and routing; removed
  topics and subscriptions from formal protocol specificatons.

2005/05/28 - 0.9a0:

- Refactored the protocol using new framing model and protocol class/method
  architecture.  First provisional draft specification.

Copyright Notice
================

This document is copyright (c) 2005 JPMorgan Inc.

Authors
=======

AMQP was designed by Pieter Hintjens with help from John O'Hara, Mark Atwell,
and others. This document was written by Pieter Hintjens.  The diagrams were
drawn by Gustavo Broos.

Abstract
========

We propose a wire-level protocol and a set of formal semantics for building
large scale and high-performance middleware architectures.  AMQP/0.9 is the
third major design iteration of the AMQ protocol.

Introduction
************

Purpose and Goals of AMQP/Fast
==============================

AMQP/Fast is designed to be the high-performance member of the AMQP
family of standardised general-purpose message-oriented middleware
wire-level protocols.

A "wire-level protocol" can be a simple as a "this is how we define
packets on the wire". AMQP/Fast is more ambitious than that. It tries to
provide formal answers to a series of questions that we must answer in
order to implement useful messaging between protocols.

We want to be able to:

- Construct logical "messages" out of data of any type and size, and
  move these message efficiently across a network of clients and
  servers.

- Provide queueing and routing semantics for messages so that the
  behaviour of the server is unambiguously defined by the protocol.

- Define explicit operational semantics for common messaging models,
  including JMS, file transfer, and streaming, at the protocol level.

- Define quality-of-service levels semantics so client applications can
  explicitly choose appropriate tradeoffs between speed and reliability.

- Create a fully-asynchronous network so that messages can be queued
  when clients are disconnected, and forwarded when they are connected.

- Allow interoperation and bridging with other middleware systems.

- Allow implementation in any language, on any kind of hardware.

- Avoid license and patent concerns that might hinder adoption of the
  protocol as a standard.

- Allow the marshalling layers of protocol implementations to be easily
  generated using code generation technology.

- Allow the creation of an abstract network where services, and their
  data, can move around the network opaquely to the clients.

Intended Operating Lifespan
===========================

AMQP/Fast is designed to give a useful lifespan of 50 years or more. Our
goal is that an AMQP/Fast peer will be able to operate continuously
with no upgrades or incompatibility for at least this duration, without
requiring "legacy support". One should be able to build a client or
server into physical infrastructure.

While the protocol version may and will change, the protocol mechanics
(framing, method structures, etc.) must operate unchanged for the full
intended lifespan of the protocol, allowing full and perfect forwards
compatability with all future versions of the protocol.

We have applied "Moore's Law" - the theory of exponential growth of
capacity of technology - to all capacity limits to identify and
eliminate potential future bottlenecks, specifically for:

1. Message sizes: the largest messages (files) are today around 20GB. We
   expect this to grow by 50% per year, reaching the limit defined by a
   64-bit size in 50 years.

2. Frame sizes: IPv4 is limited to 64KB frames, IPv6 to 4GB frames.
   Ethernet itself is limited to 12000 byte frames due to its CRC
   algorithm. We expect the maximum networking frame size to grow by
   50% per year, in large leaps. We will thus reach the limit of
   64-bit sized packets in 50 years.

3. Protocol classes and methods: the current protocol defines about
   ten classes and about ten or fewer methods per class.  We
   expect that new versions of the protocol will add classes and
   methods at a constant rate of 1 to 5 per year. The limit of 255
   classes and methods per class should be sufficient to last until
   2055.

4. Channels: the limit of 4G channels allows growth of 50% per year
   from an estimated usage of 10 channels per connection today.

5. Timestamps: we use 64-bit time-stamp values.

An Extensible Functionality
===========================

AMQP/Fast is intended to be extensible in several directions, including
new directions totally outside the scope of the protocol as it is
designed today.  These are the aspects of AMQP/Fast that have been
deliberately designed to be extensible (in order of increasing
generality and power):

1. Adding new types of exchange to server implementations.
2. Adding new properties to content domains.
3. Adding new arguments to methods.
4. Adding new methods to classes.
5. Adding new content domains.
6. Adding new method classes.
7. Adding new frame types.
8. Adding new protocols.

All of these should be feasible while maintaining full backwards
compatability with existing implementations.

Ease of Implementation
======================

Our goal is to achieve some of the cost-benefit ratio of protocols such
as SMTP and HTTP, where a simple client can be trivial to build but a
full client can be very sophisticated. Keeping AMQP/Fast accessible to
simple clients is possible if we hold to these design rules:

1. The use of all complex functionality (e.g. more sophisticated data
   types or structures) must be optional.

2. The protocol must be able to operate entirely synchronously, since
   an asynchronous model - though efficient and reliable - is a barrier
   for simple implementations.

3. The protocol must be formally defined so that significant parts of
   a client or server protocol interface can be mechanically generated
   using code generation techniques.

Guidelines for Implementors
===========================

- We use the terms MUST, MUST NOT, SHOULD, SHOULD NOT, and MAY as
  defined by IETF RFC 2119.

- We use the term "the server" when discussing the specific behaviour
  required of a conforming AMQP/Fast server.

- We use the term "the client" when discussing the specific behaviour
  required of a conforming AMQP/Fast client.

- We use the term "the peer" to mean "the server or the client".

- All numeric values are decimal unless otherwise indicated.

- Protocol constants are shown as uppercase names. AMQP/Fast
  implementations SHOULD use these names when defining and using
  constants in source code and documentation.

- Property names, method arguments, and frame fields are shown as
  lowercase names.  AMQP/Fast implementations SHOULD use these names
  in source code and documentation.

- Protocol methods are defined using a formal grammar (ASL) as defined
  in AMQ RFC xxx [still to be defined].

Glossary of Terms
=================

These terms have significance within the context of AMQP:

Connection:
    A network connection, e.g. a TCP/IP socket connection.
Channel:
    A bi-directional stream of communications between two AMQP/Fast
    peers. Channels are multiplexed so that a single network connection
    can carry multiple channels.
Client:
    The initiator of an AMQP/Fast connection or channel.  AMQP/Fast is
    not symmetrical.  Clients produce and consume messages while servers
    queues and route messages.
Server:
    The party that accepts client connections and implements the AMQ
    queueing and routing functions.
Peer:
    Either party in an AMQP/Fast connection.  An AMQP/Fast connection
    involves exactly two peers (one is the client, one is the server).
Frame:
    A formally-defined package of connection data.  Frames are
    always written and read contiguously - as a single unit - on the
    connection.
Extended_Frame:
    A frame capable of using more channels and carrying more data than
    a normal frame.
Meta_Frame:
    A specific type of frame that has the sole purpose of indicating
    that the next frame will be an extended frame.
Protocol_Class:
    A collection of methods that deal with a specific type of
    functionality.
Method:
    A specific type of frame that passes instructions from one peer
    to the other.
Content:
    Application data passed from client to server and from server to
    client.  AMQP/Fast content can be structured into multiple parts.
Header:
    A specific type of frame that describes content.
Body:
    A specific type of frame that contains raw application data.
    Body frames are entirely opaque - the server does not examine
    or modify these in any way.
Trace_Frame:
    A specific type of frame that carries information to a "trace
    handler", an abstract processing unit that may be embedded in an
    peer.
Message:
    A specific type of content, the application-to-application message.
Exchange:
    The entity within the server which receives messages from producer
    applications and decides how to forward these to queues within the
    server.
Queue:
    A named entity that holds messages and forwards them to consumer
    applications.  Queues can take many forms: transient, persistent,
    private, shared, etc.
Durable:
    A server resource that survives a server restart.
Transient:
    A server resource that is wiped or reset after a server restart.
Persistent:
    A message that the server holds on reliable disk storage and
    should not lose after a server restart.
Non-persistent:
    A message that the server holds in memory and may lose after a
    server restart.
Consumer:
    A client application that requests messages from a queue.
Producer:
    A client application that publishes messages to an exchange.
Virtual_host:
    A collection of exchanges, queues, consumers, and associated objects.
    Virtual hosts are independent server domains that share a common
    authentication and encryption environment.  I.e. the client
    application choses the virtual host after logging in to the server.
Realm:
    A set of server resources (exchanges and queues) covered by a single
    security policy and access control.  Applications ask for access
    rights for specific realms, rather than for specific resources.
Streaming:
    The process by which the server will send messages to the client at
    a pre-arranged rate.
Staging:
    The process by which a peer will transfer a large message to a
    temporary holding area before formally handing it over to the
    recipient.  This is how AMQP/Fast implements restartable file
    transfers.
Out-of-band_transport:
    The technique by which data is carried outside the network
    connection.  For example, two peers can carry frames across a
    TCP/IP connection and then switch to using shared memory if they
    discover they are on the same system.
Zero_copy:
    The technique of transferring data without copying it to or from
    intermediate buffers.  Zero copy requires that the protocol allow
    the transfer of data as opaque blocks, which AMQP/Fast does.
Assertion:
    A condition that must be true for processing to continue.
Exception:
    A failed assertion, handled by closing either the channel or the
    connection.

These terms have no special significance within the context of AMQP:

Topic:
    Usually a means of distributing messages; AMQP implements topics
    using one or more types of exchange.
Subscription:
    Usually a request to receive data from topics; AMQP implements
    subscriptions as queues.

Limitations
===========

These limitations are formally part of the AMQP/Fast specifications:

- Number of channels per connection, normal frames: 255.
- Number of channels per connection, extended frames: 64K-1.
- Size of a short string: 0 to 255 octets.
- Size of a long string: 0 to 64K-1 octets.
- Size of a normal frame: 0 to 64K-1 octets.
- Size of an extended frame: 0 to 2e64-1 octets.
- Number of possible content domains: 255.
- Number of protocol classes: 255 per peer.
- Number of methods: 255 per protocol class.

Design Proposal
***************

Data Type Representation
========================

Goals and Principles
--------------------

We use a small set of basic data types that are guaranteed to work on
all platforms and be easily implemented in all languages. More
sophisticated data types can be packaged using the basic AMQP/Fast data
types.

Formal Grammar for AMQP/Fast Fields
-----------------------------------

This formal grammar defines the AMQP/Fast data types:

    amqp-field          = BIT
                        / OCTET
                        / short-integer
                        / long-integer
                        / long-long-integer
                        / short-string
                        / long-string
                        / time-stamp
                        / field-table
    short-integer       = 2*OCTET
    long-integer        = 4*OCTET
    long-long-integer   = 8*OCTET
    short-string        = OCTET *string-char
    string-char         = %d1 .. %d255
    long-string         = short-integer *OCTET
    time-stamp          = long-long-integer
    field-table         = long-integer *field-value-pair
    field-value-pair    = field-name field-value
    field-name          = short-string
    field-value         = 'S' short-string
                        / 'L' long-string
                        / 'I' signed-integer
                        / 'D' decimal-value
                        / 'T' time-stamp
                        / 'F' field-table
    signed-integer      = 4*OCTET
    decimal-value       = decimals long-integer
    decimals            = OCTET

Guidelines for implementors:

- A peer MUST support all the above data types.

Integers
--------

AMQP/Fast defines these integer types:

- Bit values.
- Unsigned octet (8 bits).
- Unsigned short integers (16 bits).
- Unsigned long integers (32 bits).
- Unsigned long long integers (64 bits).

Integers and string lengths are always unsigned and held in network byte
order. We make no attempt to optimise the case when two low-high systems
(e.g. two Intel CPUs) talk to each other.

Guidelines for implementors:

- Implementors MUST NOT assume that integers encoded in a frame are
  aligned on memory word boundaries.

Bits
----

Bits are accumulated into whole octets.  When two or more bits are
contiguous in a frame these will be packed into one or more octets,
starting from the low bit in each octet.  There is no requirement that
all the bit values in a frame be contiguous, but this is generally
done to minimise frame sizes.

Strings
-------

All strings are variable-length and represented by an integer length
followed by zero or more octets of data.  AMQP/Fast defines these
string types:

- Short strings, stored as a 1-octet length followed by zero or more
  octets of data.  Short strings are capable of carrying UTF-8 data,
  and may not contain binary zero octets.  In the current version of
  the protocol short strings may only contain US-ASCII (ISO-8859-1)
  characters.

- Long strings, stored as a short integer length followed by zero or
  more octets of data.  Long strings can contain any data.

Timestamps
----------

Time stamps are held in the 64-bit POSIX time_t format with an accuracy
of one second.  By using 64 bits we avoid the future wraparound issues
associated with the POSIX 31 and 32-bit values.

Field Tables
------------

Field tables are long strings that contain name-value pairs. Each
name-value pair is a structure that provides a field name, a field type,
and a field value. A field can hold a tiny text string, a short binary
string, a long signed integer, a decimal, a date and/or time, or another
field table.

Guidelines for implementors:

- Field names MUST start with a letter, '$' or '#' and may continue with
  letters, '$' or '#', digits, or underlines, to a maximum length of 128
  characters.

- A peer SHOULD validate field names.

- Specifically and only in field tables, integer values are signed (31
  bits plus sign bit).

- Decimal values are not intended to support floating point values, but
  rather business values such as currency rates and amounts. The
  'decimals' octet is not signed.

- A peer MUST handle duplicate fields by using only the first instance.

As a convention when documenting values allowed in field tables we will
use this syntax:

    Name=(Type)Content

For example:

    EXPIRATION-TIME=(Time)
    IDENTIFIER=(String)01-ABCD-9876
    MAXIMUM-SIZE=(Integer)

Negotiating a Connection
========================

Goals and Principles
--------------------

Negotiation means that one party in a discussion declares an intention
or capability and the other party either acknowledges it, modifies it,
or rejects it. In AMQP/Fast, we negotiate a number of specific aspects
of the protocol:

1. The actual protocol and version.
2. Encryption arguments and the authentication of both parties.
3. Operational constraints.

Protocol and Version
--------------------

An AMQP client and server agree on a protocol and version using this
negotiation model:

1. The client opens a new socket connection on a well-known or
   configured port and and sends an initiation sequence consisting
   of the text "AMQP" followed by a protocol ID (2 decimal digits) and
   a protocol version (2 decimal digits holding the high and low version
   numbers).

2. The server either accepts or rejects the initiation sequence.  If
   it rejects the initiation sequence it closes the socket. Otherwise it
   leaves the socket open and implements the protocol accordingly.

Protocol grammar:

    protocol-header     = 'AMQP' protocol-id protocol-version
    protocol-id         = protocol-class protocol-instance
    protocol-class      = OCTET
    protocol-instance   = OCTET
    protocol-version    = protocol-major protocol-minor
    protocol-major      = OCTET
    protocol-minor      = OCTET

For AMQP/Fast these are the correct values:

- protocol-class = 1 (this is the class of all AMQP/Fast protocols)
- protocol-instance = 1 (this is the instance of AMQP/Fast over TCP/IP)
- protocol-major = 0
- protocol-minor = 9

    +---+---+---+---+---+---+---+---+
    |'A'|'M'|'Q'|'P'| 1 | 1 | 0 | 9 |
    +---+---+---+---+---+---+---+---+
                8 octets

Guidelines for implementors:

- An AMQP server MAY accept multiple AMQP protocols including AMQP/Fast
  on the same socket.

- An AMQP server MAY accept non-AMQP protocols such as HTTP.

- An AMQP server MUST accept the AMQP/Fast protocol as defined by
  class = 1, instance = 1.

- If the server does not recognise the first 4 octets of data on the
  socket, or does not support the specific protocol version that the
  client requests, it MUST close the socket without sending any
  response back to the client.

- An AMQP client MAY detect the server protocol version by attempting
  to connect with its highest supported version and decreasing this if
  the server rejects the connection.

- An AMQP server MUST NOT ban or delay a client that requests an
  unsupported protocol version.

- An AMQP server MAY ban or delay a client that requests unknown
  protocol IDs.

The protocol negotiation model is compatible with existing protocols
such as HTTP that initiate a connection with an constant text string,
and with firewalls that sniff the start of a protocol in order to decide
what rules to apply to it.

Encryption and Authentication
-----------------------------

AMQP/Fast uses the SASL architecture for security. SASL encapsulates
TLS, GSSAPI, Kerberos, and other encryption and authentication
technologies.

Security is negotiated between server and client as follows:

1. The server sends a challenge to the client. The challenge lists the
   security mechanisms that the server supports.
2. The client selects a suitable security mechanism and responds to the
   server with relevant information for that security mechanism.
3. The server can send another challenge, and the client another
   response, until the SASL layer at each end has received enough
   information.
4. The server can now use the selected security mechanism and
   authenticated client identity to perform access controls on its
   data and services.

The "relevant information" is an opaque binary blob that is passed
between the SASL layers embedded in the client and in the server.

SASL gives us the freedom to replace the security libraries with better
(more secure or faster) technologies at a later date without modifying
the protocol, the client, or the server implementations.

Client and Server Limits
------------------------

The protocol allows the client and server to agree on limits to ensure
operational stability. Limits allow both parties to pre-allocate key
buffers, avoiding deadlocks. Every incoming frame either obeys the
agreed limits, so is "safe", or exceeds them, in which case the other
party is faulty and can be disconnected.

Rather than fix the limits in the protocol, we negotiate them. This lets
us tune the protocol dynamically for different types of efficiency, e.g.
minimal memory consumption vs. maximum performance.

Limits are negotiated to the lowest agreed value as follows:

1. The server tells the client what limits it proposes.
2. The client can respond to lower the limits for its connection.

Identification and Capabilities
-------------------------------

During the negotiation of limits the peers MUST pass this mandatory
information about themselves:

- The specific content domains that they support (see later).

And the peers MAY pass this optional information about themselves:

- Their public product name, for logging and tracking.
- Their version number.
- The platform they are running on.

For security reasons, all these pieces of information are optional and a
highly-security conscious peer MAY choose to provide some or none of
them. A peer MUST NOT use this information to adapt its behaviour.

Multiplexing and Pipelining
===========================

Goals and Principles
--------------------

Connection multiplexing allows multiple threads of communication to
share a single socket connection. This is valuable when threads are
short-lived since the cost of opening and closing TCP/IP sockets can be
relatively high.

Pipelining means that each peer can send frames asynchronously without
waiting for the recipient to acknowledge each one. This is an important
facility because it greatly improves performance.

Multiplexing Design
-------------------

AMQP/Fast uses the concept of "channel" to carry bi-directional streams
of communication between peers.  AMQP/Fast provides methods to open,
use, and close channels.  Large messages are broken into smaller frames
so that channels get fair (round-robin) use of the socket connection.

Pipelining Design
-----------------

Our design is based on these principles:

1. Most operations succeed.  We can thus optimise traffic in many cases
   by stating that "no response" means "successful".

2. Channels represent serial streams, where the order of methods and
   their acknowledgements is stable.

3. In general we want to send data as fast as possible in an asynchronous
   fashion.  Where necessary, we can implement windowing and throttling
   at a higher level.

The recipient indicates an error by closing the channel. The close
method includes the ID of the method that caused the error.  Methods on a
specific channel are processed strictly in order.

The Framing Design
==================

Goals and Principles
--------------------

Framing is the part of the protocol where we define how data is sent "to
the wire".  The framing design used in AMQP/Fast is designed to be compact,
easy and very fast to parse, extensible, and robust.

The key parts of this design are:

- How the transport layer carries data.
- How we delimit frames on the connection.
- How we multiplex frames on the connection.
- How different types of frame carry data.
- How we handle out-of-band transport.
- How we support future protocols (e.g. IPv6).

The Transport Layer
-------------------

We assume a reliable stream-oriented network transport layer (TCP/IP or
equivalent). If an unreliable transport layer is used, we assume that
the AMQP/Fast frames would be wrapped with additional traffic-control
information such as windowing.

AMQP/Fast explicitly excludes any support for traffic control but does
not disallow this to be implemented in an additional layer. We may at a
future point provide protocol wrappers that wrap AMQP/Fast frames with
traffic management to allow reliable transfer over multicast and
point-to-point UDP.

Delimiting Frames
-----------------

TCP/IP is a stream protocol, i.e. there is no in-built mechanism for
delimiting frames. Existing protocols solve this in several different
ways:

- Sending a single frame per connection. This is simple but adds
  considerable overhead to the protocol due to the opening and closing
  of connections.

- Adding frame delimiters to the stream. This is used in protocols
  such as SMTP but has the disadvantage that the stream must be parsed
  to find the delimiters. This makes implementations of the protocol
  slow.

- Counting the size of frames and sending the size in front of each
  frame. This is the fastest approach, and our choice.

Each frame is thus sent as a "frame header" which contains the frame
size, followed by a "frame body" of the specified size (the header is
not counted). Frames can carry methods and other data. To read a frame
we use this logic:

1. Read and decode the frame header to get the size of the frame body.
2. If the frame body is larger than the agreed limit, close the
   connection with a suitable error reply code.
3. Otherwise, read the specified number of octets into a frame buffer.
4. Decode the frame buffer as needed.

To write a frame we use this logic:

1. Marshal the frame buffer as needed.
2. Encode the length of this buffer into a frame header and write it.
3. Write the frame buffer contents, being the frame body.

Guidelines for implementors:

- A peer MUST write the frame body immediately after the frame header
  unless the body size is zero, or out-of-band transport is being used.

The Frame Header Wire Format
----------------------------

All frames start with a 4-octet header composed of a one-octet type field
a 1-octet channel field and a short size field:

    0      1         2             4               size+4
    +------+---------+----------+  +------------+  +-----------+
    | type | channel |   size   |  | frame body |  | frame-end |
    +------+---------+----------+  +------------+  +-----------+
     octet    octet     short       size-1 octets      octet

AMQP/Fast defines these types of frame:

- Type = 1, "METHOD": method frame.
- Type = 2, "HEADER": content header frame.
- Type = 3, "BODY": content body frame.
- Type = 4, "OOB-METHOD": out-of-band method frame.
- Type = 5, "OOB-HEADER": out-of-band band header frame.
- Type = 6, "OOB-BODY": out-of-band body frame.
- Type = 7, "TRACE": trace frame.
- Type = 8, "HEARTBEAT": heartbeat frame.
- Type = 9, "META": meta frame.

The channel number is 0 for all frames which are global to the connection
and 1 .. 255 for frames that refer to specific channels (1 .. 64K-1 in
extended frames).

The size field includes the end field, a single octet with the value
%xCE.

Guidelines for implementors:

- If a peer receives a frame with a type that is not one of these defined
  types, it MUST treat this as a fatal protocol error and close the
  connection without sending any further data on it.

- All frames include the frame-end octet. The minimum value for the size
  field is 1.

- When a peer reads a frame it SHOULD check that the frame-end is valid
  before attempting to decode the frame.  If the frame-end is not valid
  it MUST treat this as a fatal protocol error and close the connection
  without sending any further data on it.  It SHOULD log information
  about the problem.

Method Frames
-------------

The method frame design is intended to be fast to read and write, and
compact when only partly filled.

Method frames (also called "methods" in the following discussion, for
brevity) are invariant. They contain no conditional or repeated fields.
Methods are constructed out of a constant series of AMQP/Fast data
fields (bits, integers, strings and string tables). Thus the marshalling
code can be easily generated, and can be very rapid. In the OpenAMQ
server and clients this code is generated directly from the protocol
specifications.

Content Frames
--------------

Certain methods signal to the recipient that there is content arriving
on the connection.  Simple content consists of a header frame followed
by zero or more body frames.  Structured content may consist of a series
of header frames followed by the appropriate body frames.

Looking at the frames as they pass on the wire, we might see something
like this:

    [method]
    [method] [header] [body] [body]
    [method] [header] [body] [body]
    [trace]
    [method]
    ...

There is a specific reason for placing content body in distinct frames
as compared to including the header and body in the method. We want to
support "zero copy" techniques in which content is never marshalled or
encoded, and can be sent via out-of-band transport such as shared memory
or remote DMA.

Guidelines for implementors:

- The channel number in content frames MUST NOT be zero.

- Content frames on a specific channel form an strict list. That is,
  they may be mixed with frames for different channels, but two contents
  may not be mixed or overlapped on a single channel.

Out-of-Band Transport Frames
----------------------------

Method, content header, and content body frames can be sent using
out-of-band transport. The frame header is sent on the normal connection
but the frame body is sent via another mechanism. The specific
out-of-band transport used, and its configuration, is defined when a
channel is opened.

Trace Frames
------------

Trace frames are intended for a "trace handler" embedded in the
recipient. The significance and implementation of the trace handler
is implementation-defined.

Guidelines for implementors:

- Both server and client MAY send trace frames at any point in the
  connection after protocol negotiation and before a Connection.Close
  method.

- If the recipient of a trace frame does not have a suitable trace
  handler, it MUST discard the trace frame without signalling any error
  or fault.

- The semantics and structure and of trace frames including the channel
  number are not formally defined by AMQP/Fast and implementations MUST
  NOT assume any interoperability with respect to trace frames unless
  and until formal standards are defined for these.

Heartbeat Frames
----------------

Although TCP/IP guarantees that data is not dropped or corrupted it can
be slow to detect that a peer process has gone "offline". The heartbeat
frame allows peers to detect network failure rapidly. This frame has no
body.

The peers negotiate heartbeat parameters at the start of a connection.

Guidelines for implementors:

- The channel number in a heartbeat frame MUST be zero.

- An AMQP peer MUST handle a non-zero channel number as a connection
  exception.

Meta Frames
-----------

Meta frames are used to send frames of 64KB or larger (the size of a
normal frame body is limited to 64KB-1 octets), and to use channel
numbers above 255. Meta frames are designed to support high-capacity
out-of-band transports and IPv6 (which has a 4GB frame limit, although
ethernet still cannot handle frames larger than 12000 bytes).

Extended Frame Header Wire Format
---------------------------------

A meta frame is followed by an "extended frame header" with a short
integer channel number and a long long size:

    0      1   2   4
    +------+---+---+
    | META | 0 | 0 |
    +------+---+---+
    0       1       2         4           12
    +-------+-------+---------+------------+ +-------------- - -
    | type  | flags | channel | frame-size | | frame body...
    +-------+-------+---------+------------+ +-------------- - -
      octet   octet   short     long long     'frame-size' octets

Guidelines for implementors:

- The meta frame channel number and size MUST be zero.

- The sender MUST NOT send intervening frames between the meta frame and
  the extended frame.

- After a meta frame all frame types are valid except a meta frame.

- The "flags" octet is reserved for future use and MUST be be set to
  zero.

- The frame body is not affected in any way by the presence or absence
  of a meta frame, except that the frame body MAY be larger than 64KB
  after a meta frame.

The Class/Method Model
======================

Goals and Principles
--------------------

AMQP/Fast allows two peers to connect and pass data. There are
several types of data and several ways of exchanging this data. The
combinations rapidly get complex. Worse, it is not always clear which
replies depend on which requests, since the passing of data is often
asynchronous, pipelined, and generally hard to follow.

Rather than attempt to formalise the exact flow of data from client to
server and back, we use a metaphor taken from the software world,
namely the object-oriented metaphor of "classes" and "methods".

We build this as follows:

1. Each peer supports a set of classes.  These classes have names
   and cover specific areas of functionality.  A peer can support
   all the AMQP/Fast classes, or a subset of them.  Since the
   operational functionality of the protocol depends on the classes,
   the operational functionality of a particular implementation is
   clear to see.

2. Each class supports a set of methods.  These methods have names
   and provide specific functionalities.  A peer can support all
   the AMQP/Fast methods in a particular class, or a subset of them.
   As with classes, this unambiguously defines the operational
   functionality of a peer.

3. The methods are either synchronous or asynchronous.  A synchronous
   method replies immediately.  An asynchronous method replies at some
   later time and may send multiple replies.  A method "replies" by
   invoking the appropriate method in the requesting peer.  A request
   and reply always involves two methods: one to implement the request
   and one to implement the reply.

4. Each method is either a client method, server method, or provided by
   both client and server). A particular method in a particular class
   always has an identical behaviour whether it is in a "client" or
   "server".

5. A peer can thus combine the roles of client and server in various
   degrees by implementing the necessary classes and methods. At any
   point the operational functionality of the peer can be unambiguously
   defined by the methods that it supports.

Client and Server Roles
------------------------

The AMQP/Fast protocol is quasi-symmetrical with client-initiated
connections. We can visualise the protocol as governing two levels of
connection - network connections (sockets) and virtual connections
(channels):

    +-----------+-----------+-----------+-----------+
    |  channel  |  channel  |  channel  |  channel  |
    +-----------------------------------------------+
    |                    socket                     |
    +-----------------------------------------------+

Each peer involved in a connection (socket and channel) is strictly
a client or a server.  The client initiates the connection and the
server accepts the connection.  This applies both to network and to
virtual channel connections:

1. The client requests the server to open connection.
2. The server responds and the peers agree on connection parameters.
3. The client and the server exchange information.
4. Either peer requests to close the connection.
5. The other peer responds and they negotiate the shut-down.

AMQP/Fast forsees two main architectures that use this model:

1. Client-server, in which the same peer acts as client in all
   connections.  This is the "usual" architecture.
2. Peer-to-peer, in which one peer acts as client in the network
   connection but the peers can take either role in the virtual
   connection.

The peer-to-peer architecture requires that both peers can act as both
client and server.

Guidelines for implementors:

- A peer MUST support at least either the client or server role.

- A peer MAY support both roles.

Method Frame Body Wire Format
-----------------------------

A method frame body has this format:

    0       1        2          4
    +-------+--------+----------+-------------- - -
    | class | method | reserved | arguments...
    +-------+--------+----------+-------------- - -
      octet   octet     short

- The class and method are protocol-constant values.

- The reserved must be zero in this version of the protocol.

- The arguments are a set of AMQP/Fast fields that specific to each
  method.

The Abstract Content Model
==========================

AMQP/Fast uses an abstract content model that has these goals and
features:

- It supports tree-structured content in which each content can
  contain further content ("child-content"), to any level.

- It provides multiple content types to allow optimal encoding for
  different applications.

- It supports content bodies of any size from zero octets upwards.

- Content body can be read and written directly from application memory
  with no formatting or copying ("zero copy").

- Content headers are sent before content body so that the recipient can
  selectively discard content that it does not want to process.

- Content body is sent in separate frames to support the AMQP/Fast
  channel multiplexing model.

Formal Grammar for Content Frames
---------------------------------

This is the formal grammar for the AMQP/Fast content model:

    content         = header-frame child-content *body-frame
    header-frame    = HEADER channel frame-size header-payload frame-end
    channel         = octet
    frame-size      = short-integer
    header-payload  = class weight body-size
                      property-flags property-list
    content-class   = OCTET
    content-weight  = OCTET
    body-size       = long-long-integer
    property-flags  = 15*BIT %b0 / 15*BIT %b1 property-flags
    property-list   = amqp-field
    child-content   = weight*content
    body-frame      = BODY channel frame-size body-payload frame-end
    body-payload    = *OCTET
    frame-end       = OCTET %xCE

Content Header Frame Wire Format
--------------------------------

A content header frame has this format:

    0        1        2           10               12
    +--------+--------+-----------+----------------+------------- - -
    | domain | weight | body size | property flags | property list...
    +--------+--------+-----------+----------------+------------- - -
      octet    octet    long long   short int        remainder...

- The content class specifies the set of properties (with a predefined
  syntax and semantics) that the header frame may hold. The standard
  AMQP/Fast content classes are defined later.

- The weight field specifies whether the content is structured or not.
  Unstructured content has a weight of zero.  Structured content has a
  weight of 1 to 255.  This is the number of child-contents that the
  content contains.

- The body size is a 64-bit value that defines the total size of the
  body content.  It may be zero, indicating that there will be no
  body frames.

- The property flags are a bit array that indicates the presence or
  absence of each property value.

- The property values are class-specific data fields.  The properties
  are formatted as AMQP/Fast data types (integers, strings, field tables).

- Bit properties are indicated ONLY by their respective property flag
  (1 or 0) and are never present in the property list.

The set of properties for a content class can be any size. The first
two properties for all content classes are the content MIME type and the
content encoding. Following these, each content class has a specific
set of properties defined in a strict order from most to least
significant.

As an example, we take an imaginary content class "D" which has three
properties, T, E and C. Imagine a simple content that has properties E
and C but not T. The header frame will be formatted thus:

      domain   weight   body size   property flags   property list
    +--------+--------+-----------+----------------+-------+
    |   D    |   0    |    nnn    | 011...         | E | C |
    +--------+--------+-----------+----------------+-------+

The property flags are ordered from high to low, the first property
being indicated by bit 15, and so on:

     15  14  13  12  11  10   9   8   7   6   5   4   3   2   1   0
    +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
    | T | E | C |                                               | 0 |
    +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+

The property flags can specify more than 16 properties. If bit 0 is set,
it indicates that a further property flags field follows. There can be
several property flag fields in succession:

     15  14  13  12  11  10   9   8   7   6   5   4   3   2   1   0
    +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
    |                                                           | 1 |
    +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
    |                                                           | 1 |
    +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
    |                                                           | 0 |
    +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+

Content Body Frame Wire Format
------------------------------

A content body frame has this format:

    +-----------------------+-----------+
    | Opaque binary content | frame-end |
    +-----------------------+-----------+

Where the size of the frame is defined in the frame header.

Structured Content
------------------

The "weight" field in the content header frame indicates whether the
content has child-contents or not. In the simplest case (weight = zero)
content consists of a header frame plus zero or more body frames:

    [header] [body] [body]...

If the content has child-contents, these are inserted between the
header and body of the parent content:

    [parent-header]
        [child-header] [child-body] ...
    [parent-body]

This organisation means that a principal content (e.g. a very large
video file) can have child content (e.g. subtitles and menus) which will
be sent before the main content. If a more specific order of content is
needed, it can be done by defining a top-level content that has no body,
only a header.

Multiplexed Channel Content
---------------------------

Content consists of one or more frames. The recipient of content that
has been sent in multiple frames can unambiguously reconstruct the
content by implementing the content grammar defined above, on a
per-channel basis.

Guidelines for implementors:

- Content frames MUST be sent on a single channel, and MUST NOT be
  intermixed with non-content frames on the same channel (method, trace,
  and heartbeat frames).

- Content frames MUST NOT be intermixed with frames of a different
  content.

- A peer MUST NOT make any assumptions about the minimum or maximum size
  of a body frame except that the minimum size is zero bytes and the
  maximum size is the smallest of either the negotiated maximum frame
  size or the remaining expected of content body (body-size minus amount
  of body data received so far).

- Empty body frames are permitted but a peer may not assign any special
  meaning to these: it MUST discard them with no side-effects of any kind.

Content Classes
----------------

The content class is not intended to provide data typing or encoding
information. Its purpose is to allow functional clarity in the property
sets and methods that we define per class of message.

Content classes turn AMQP/Fast from an abstract content-carrying
protocol into a functional tool by providing data and operations that
are directly and clearly mapped to the kinds of API that an application
programmer needs to see.

AMQP/Fast defines these standard content classes:

- Basic: the content is an untyped message.
- JMS: the content is a JMS-compliant message.
- File: the content is a persistent file.
- Stream: the content is a streamed message.

Note that the ID of each content class is the same as the ID of the
protocol class (Basic, JMS, File, Stream) that supports it.

Each class has a specific access model, though they all share the basic
AMQP queueing and routing mechanisms including topic multipliers. These
classes are defined in the next sections.  This model lets us extend
AMQP/Fast into new functional areas by defining new content classes and
appropriate methods, without breaking interoperability.

Guidelines for implementors:

- The client MAY support any or all of the defined content classes.

- The server MUST support at least one of the defined content classes.

Content Type and Encoding
-------------------------

AMQP/Fast enforces MIME-compliance on all content classes using two
standard properties that are the first defined in all content classes:

- ContentType (short string) - the MIME content type of the message body.
  The default value is "application/octet-stream".

- ContentEncoding (short string) - The content encoding of the message
  body. The default value is "binary".

Guidelines for implementors:

- The server SHOULD not modify the ContentType and ContentEncoding
  for content moved between producing and consuming clients.

- The server MAY set the ContentType and ContentEncoding properties
  for content produced by the server or by applications embedded in
  the server.

The Abstract Routing Model
==========================

AMQ defines an abstract model which all AMQP servers and clients must
adhere to.  The abstract routing model provides a formal and run-time
configurable definition of how messages are stored and forwarded.

Introduction to the Abstract Routing Model
------------------------------------------

Restatement of Goals
....................

AMQ is designed to carry data ("messages") between applications
("clients") written in arbitrary languages and running on arbitrary
platforms spread across a fully-addressable network.

AMQ is designed to support many different types of application, each
with its own requirements as to:

- Message size, from zero bytes up to many gigabytes.
- Number of clients, from one up to many thousands.
- Volume of messages crossing the network, from a few per day to hundreds
  of thousands per second.
- Critical latency of a message, from a millisecond to an hour.
- Reliability of delivery, from "as reliable as possible, whatever the
  cost" to "best-effort delivery".
- Use of network, memory, CPU, and disk resources.

The same AMQ architecture must be able to work efficiently on a small
embedded message router, as well as on a supercluster with dozens of
CPUs and tens of gigabytes of memory.

General Design Method
.....................

Our design method relies on these principles:

1. Give the functional architect (the person responsible for designing
   an AMQ network, using AMQ technology) as much control as possible over
   the choices and tradeoffs required for a specific functional scenario.

2. Design the architecture using abstract general models that the
   functional architect can combine as needed.

That is: rather than embed assumptions and decisions in the architecture
we create abstract general-purpose constructs that can be reused in
scenarios that we do not pretend to fully understand ourselves.

Main Insights
.............

The main insights that led to our design (which is the third major
refactoring of the AMQ protocol) were:

1. Subscriptions can be designed as a special case of queues.  That is,
   subscriptions and queues can be designed to support exactly the same
   set of properties and access methods.  In the topic/subscription
   model subscriptions are simply queues that have a special matching
   technique and a private name space.

2. Matching can be abstracted.  That is, everything from matching on a
   specific destination name to matching on an SQL-like selector can be
   abstracted into a single concept.

3. Messages can have multiple instances.  That is, a message can be in
   many queues at the same time.  We do not copy message content, rather
   we use message reference counting.  Queues store references to
   messages rather than messages.  This applies both to persistent and
   non-persistent messages.

4. Queues and matching can be loosely-coupled.  That is, while one
   client "writes to queue X" and a second client "reads from queue
   X", this can be done using a less tightly-coupled relationship than
   the obvious "producer writes to queue".  Loose coupling is often
   desirable because it allows abstraction in which one side of a
   relationship can be reorganised without affecting the other.

We use these insights to define a set of building blocks that achieve
much wider functionality than classic messaging middleware (like JMS)
at no significant extra cost.

Overall Architecture
--------------------

Classic Message-Oriented Middleware
...................................

A classic message-oriented middleware server provides two types of
service: a "queue" service in which producers and consumers interact
via shared named queues, and a "topic" service which allows clients to
subscribe to a tree of topics. Clients subscribe with citeria such as
the topic name, or a pattern representing a set of topics, e.g.
"FOREX.*". The middleware explicitly provides functionality to
subscribe, manage topics, and so on.

AMQ does not provide these services directly: the protocol and server
make no mention of "topics" or "subscriptions". Rather, AMQ provides
more general building blocks that can easily be used to build a topic
service, a queue service, a mix of the two, and new kinds of service
that fall outside the classic definitions.

Message, Queue, and Exchange
.........................

We define the "MESSAGE": a message is self-contained package of data
that passes through the server. The server will not modify or separate
the data in a message. The message is the basic unit of data transfer
between clients and servers.

We define the "QUEUE": a queue holds messages and distributes these to
clients. Queues are highly configurable objects that implement most of
the quality of service aspects of an AMQP server. Queues can be private
or shared, created by the server administrator, or by applications.
Queues can holding their messages on disk or in memory, or a combination
(working in memory and overflowing to disk when necessary). Queues can
be durable (remaining active when the server stops and restarts) or
transient (automatically deleted when no longer used).

Queues have public names, and when a client wants to get messages from a
server it says, "give me messages from queue such-and-such".

We define the "EXCHANGE": an exchange is a named object that accepts
messages and distributes them to queues. A message can exist on many
queues at once.  Clients that want to send messages to a server always
send them to a specific exchange on that server.

Basic Architecture
..................

The basic architecture is thus (showing a set of producer applications
sending messages to a set of consumer applications):

                 +--------+        +--------+
    Producers    | Client |        | Client |
                 +--------+        +--------+
                     |                  |
                     V                  V
    Messages     +----\-----Server-----/----+
        |        |     \              /     |
        |        |      +------------+      |
        |        |      |  Exchange  |      |
        |        |      +------------+      |
        |        |         /      \         |
        |        |  +-------+    +-------+  |
      __|__      |  | Queue |    | Queue |  |
      \ | /      |  +-------+    +-------+  |
       \|/       |     /              \     |
        V        +----/----------------\----+
                     |                  |
                     V                  V
                 +--------+        +--------+
     Consumers   | Client |        | Client |
                 +--------+        +--------+

Note that:

1. The producers do not write directly to the consumers' queues.
2. The consumers do not read directly from the exchange.

Queue Consumers
...............

While exchanges copy messages across queues, queues distribute messages
between consumers.  A queue can have one consumer:

        +-------+
        | Queue |
        +-------+
            |
            V
        +--------+
        | Client |
        +--------+

A queue can also have many consumers:

                        +-------+
                        | Queue |
                        +-------+
                            |
           .----------.-----------.-----------.
           |          |           |           |
           V          V           V           V
      +--------+  +--------+  +--------+  +--------+
      | Client |  | Client |  | Client |  | Client |
      +--------+  +--------+  +--------+  +--------+

The queue distributes messages to the consumers according to a
configurable traffic control model that is adjustable per consumer:

- Each consumer specifies whether it is going to acknowledge
  messages or not.  Acknowledgements improve reliability but
  slow-down performance.

- Each consumer specifies how many messages the server should
  send it in advance of acknowledgements.  This "prefetch"
  window improves latency but can mean that consumers are not
  perfectly balanced.

Given these two parameters the queue will distribute messages to the
consumers on a quasi-round-robin basis (i.e. to each consumer in turn,
but without a guarantee of visting each consumer in any particular
order).

Relationship between Queues and Exchanges
.......................................

A queue "registers" with an exchange, asking: "give me a copy of all
messages that look like this".

AMQ has a formal definition of a message which includes:

- the "destination" for the message.  This is a name that the client
  application uses.  The destination name may (or may not) map directly
  to a queue name.

- the "properties" for the message.  These are a set of fields with
  a name, type, and value.  The field properties may (or may not) be
  used by the exchange to decide which queues receive the messages.

The Exchange Design
-------------------

Exchange Types
..............

These are the basic set of exchanges that an AMQP server MAY support:

1. "fanout" routes all messages to all registered queues.  Fanout
   exchanges are used to create arbitrary routing flows within the
   server.  For example, fanout exchanges might be used to replicate
   data within a cluster.

2. "destination" routes messages on their destination name using a direct
   comparison. That is, it passes messages to queues that have
   registered with "give me all messages whose destination name is equal
   to this value".  Dest-name exchanges are used to build conventional
   point-to-point store-and-forward mechanisms.

3. "wildname" routes messages on their destination name using a
   wildcard comparison.  That is, it passes messages to queues that have
   registered with "give me all messages whose destination name matches
   this pattern".  Dest-wild exchanges are used to build conventional topic
   publish-and-subscribe mechanisms.

4. "property" routes messages on the value and presence of properties.
   That is, it passes messages to queues that have registered with
   "give me all messages with properties with these names and values".
   Prop-test exchanges are used to build content-based routing mechanisms.

5. "matcher" routes messages on a set of properties using wild-card
   comparisons.  That is, it passes messages to queues that have
   registered with "give me all messages with properties that match
   these specifications".  Prop-wild exchanges are used to build
   content-based routing mechanisms.

6. "selector" routes messages using a programmable selector statement.
   That is, it passes messages to queues that have registered with
   "give me all messages where this statement is true".  The selector
   statements are based on SQL WHERE conditions.  Selector exchanges are
   used to build content-based routing mechanisms.

7. "lossy" routes messages with random failure. That is, it passes
   messages to all registered queues, but randomly drops messages.
   Lossy exchanges are used to simulate network or processing failures.

Guidelines for implementors:

- An AMQP server MUST implement at least the fanout and dest-name
  exchanges.  An AMQP server MAY implement the other exchange types.

- An AMQP server MAY implement custom exchange types (not on this list).
  The name of a custom exchange type must start with "x-".

Multiple Registrations
......................

A queue can register with the same exchange several times:

    Register with dest-name: messages for destination X
    Register with dest-name: messages for destination Y
    Register with dest-name: messages for destination Z

The meaning of this is obvious and does not need much explaining: the
registrations are acummulated so the queue will get messages for any of
the three destination names:

         +-----------+
         | dest-name |
         +-----------+
             | | |  "messages for destinations X, Y, Z"
           +-------+
           | Queue |
           +-------+

A queue can also register with two different exchanges:

    Register with dest-name: messages for destination X
    Register with dest-name: messages for destination Y
    Register with dest-name: messages for destination Z
    Register with lossy

The meaning of this is as follows:

         +-----------+
         | dest-name |
         +-----------+
             | | |  "messages for destinations X, Y, Z"
           +-------+
           | lossy |
           +-------+
               |    "all messages"
           +-------+
           | Queue |
           +-------+

In other words, the lossy exchange steps in between the queue and the
dest-name exchange.  We could imagine several levels:

         +-----------+
         | dest-name |
         +-----------+
             | | |  "messages for destinations X, Y, Z"
         +-----------+
         | prop-name |
         +-----------+
               |    "messages with property ABC"
           +-------+
           | lossy |
           +-------+
               |    "all messages"
           +-------+
           | Queue |
           +-------+

Guidelines for implementors:

- An AMQP server SHOULD support multiple registrations but MAY restrict
  its exchange implementation to a single exchange or single registration
  per queue.

On-Demand Wiring
................

To provide maximum flexibilty we split the definition of a server's
wiring (i.e. the exchanges, queues, and relationships between them) into
three layers:

1. The set of exchange classes and how they are implemented.  This layer
   is defined in the server's software.

2. A set of pre-defined exchanges and queue templates.  This layer is
   defined in the server's configuration files.

3. The final set of exchanges, queues and registrations. This layer is
   defined at runtime by the application.

        +--------------------------+
     1  |  Software constructions  |   E.g. what exchange classes exist?
        +--------------------------+
                     |
                     V
        +--------------------------+
     2  |   Configured defaults    |   E.g. what queue templates exist?
        +--------------------------+
                     |
                     V
        +--------------------------+
     3  |     On-demand wiring     |   E.g. create exchanges & queues
        +--------------------------+
                     |
                     V
               Actual Wiring

Thus the protocol provides functionality to create and destroy exchanges
and queues, and register queues with exchanges.

Guidelines for implementors:

- An AMQP server SHOULD support queue templating in which queues
  created at runtime can be configured by reference to a
  pre-configured template.

Durable Wiring
..............

An AMQ server may hold purely transient data, but may equally hold
persistent data. In the first case we want a "clean restart" if a server
is stopped and restarted. I.e. all dynamically-created exchange and queue
definitions should be wiped. In the second case we want a "reliable
restart" so that queues keep their contents safely.

We use the concept of a "durable resource". This is a resource created
by the client application but held by the server as if it was a
configured resource. We need durable exchanges, durable queues, and
durable registrations. The opposite of "durable" is "transient".

The actual or required durability of a resource is defined as follows:

1. If a resource is part of the server's configuration, it is always
   durable.  For example, an exchange defined in the server configuration
   is always durable.

2. When a resource is created the client application may ask for it to
   be durable.  For example, a queue can be created as "durable".

3. If the client application attempts to create a dependency from a
   durable resource onto a transient resource, this causes a channel
   exception.  For example, if a client attempts to register a durable
   queue with a transient exchange the server will close the channel with
   an appropriate error.
   
Worked Examples
---------------

Reconstructing Store and Forward
................................

Let us reconstruct queue-based "store and forward" using these building
blocks:

- We create a dest-name exchange called "queue".
- We agree that queue names are equal to the destination name placed
  in each message.
- We create a set of queues and register these with the "queue" exchange.
- Publishers send their messages to the "queue" exchange, specifying a
  destination name with each message.
- The "queue" exchange does a lookup on the destination name and passes
  the message to the matching queue.
- Consumers get their messages from each named queue.

Reconstructing Topics and Subscriptions
.......................................

Let us reconstruct topic-based "subscribe and publish" using these
building blocks:

- We create a dest-wild exchange called "topic".
- We agree that queue names are subscription names.
- We create one queue per subscription and attach these queues to
  the "topic" exchange.
- Publishers send their messages to the "topic" exchange, specifying a
  destination name with each message.
- The "topic" exchange does a topic-name match on the destination name
  and passes the message to all queues that match.
- Consumers get their messages from each named queue.

Reconstructing Temporary Destinations
.....................................

- A temporary queue is simply a queue created by a client, and flagged
  for automatic deletion when the client disconnects.

- A temporary topic has no meaning: topics do not exist except as values
  on which to match.

Implementation Details
----------------------

Queue Namespaces
................

We want to provide the client application with a consistent way to
distinguish different types of queue, for example "subscriptions" and
"private reply queues". These types have no meaning to the server but to
avoid non-standard naming conventions we introduce the concept of a
"queue namespace", which is a type name. The server simply prefixes the
queue name with the namespace to create a unique internal name.

By convention in many scenarios, the namespace will be the same as the
exchange name.

Thus a publisher may send to an exchange called "queue", while the consumer
might read messages from a queue in namespace "queue".

Queue Templating and Tuning
...........................

Templating is a normalisation technique: rather than define many objects
with similar properties we define a set of templates, and then refer to
these when we define new objects. In object-oriented development this
may be called a "class".

A queue template provides different types of information:

1. Organisation and tuning: file system criteria, maximum queue size,
   dead-letter queue, etc.
2. Constraints: e.g. max consumers.

Queue templating could be used to simplify the protocol. For example it
is useful to be able to specify "this queue must be deleted when there
are no more consumers". This could be a protocol flag or it could be a
template property, understood by the server but not part of the explicit
protocol.

We have to decide which queue properties are made visible in the
protocol, and which can be configured in templates. Our choice is based
on the requirement for interoperability:

- A property that has an impact on interoperability must be visible
  in the protocol.
  
- A property that must be tunable by the administrator must be part
  of the template.

Thus the "delete this queue when there are no more consumers" must be
specified in the protocol, since it is necessary for interoperability.
The fact that a queue automatically disappears when it is no longer
needed is a vital part of some architectures. In contrast, a queue limit
such as the maximum number of messages allowed in memory before the
queue overflows onto disk is an adminisration concern, so must be part
of templated configuration.

Exchange Reference Design
.........................

We propose a reference design for the exchange concept. While this does
not form part of the protocol specification, and while each server
implementation may implement exchanges as suitable, this reference
design may help to clarify the exchange concept.

We design each exchange as a processing class in the server, with
properties and methods that allow it to operate. All exchanges have a
common abstract structure:

    +------+-------+-------------------+------------------------------+
    | Name | Class | Registration list | Class-specific properties... |
    +------+-------+-------------------+------------------------------+
                            :
                     +--------------+    +-------+
                     | Registration |--->| Queue |
                     +--------------+    +-------+
                            :
                     +--------------+    +-------+
                     | Registration |--->| Queue |
                     +--------------+    +-------+
                            :
                           ...

In some scenarios many queues will register with the same exchanges
using identical registrations. We can foresee that this will create
problems in high-volume scenarios - e.g. when several thousands of
queues register with the same exchange. Our class design therefore
normalises identical registrations, using a list of queues per
registration:

    +------+-------+-------------------+------------------------------+
    | Name | Class | Registration list | Class-specific properties... |
    +------+-------+-------------------+------------------------------+
                            :
                     +--------------+    +-------+    +-------+
                     | Registration |--->| Queue |--->| Queue |--->...
                     +--------------+    +-------+    +-------+
                            :
                     +--------------+    +-------+
                     | Registration |--->| Queue |--->...
                     +--------------+    +-------+
                            :
                           ...

Queueing and Routing Mechanisms
===============================

Goals and Principles
--------------------

The AMQP queueing and routing mechanisms define how content is moved
between an AMQP server and its clients. Our goal is to define a single
coherent queueing and routing model that can support a range of different
highly-functional content classes.

Our design is based on these elements:

- Virtual hosts: an independent collection of exchanges and queues.

- Exchanges: an exchange distributes messages across queues.

- Queues: a queue distributes messages between consumers.

- Message persistence: defining the level of assurance that a message
  will be successfully delivered despite crashes and failures.

- Message priorities: allowing messages to be delivered out-of-sequence
  according to their importance.

- Queue browsing: allowing a client to read messages in a simple
  synchronous manner.

- Acknowledgements: allowing a client to tell the server when it has
  successfully processed a message.

- Window-based flow control: allowing a client to control how many
  messages it receives

- Transactions: allowing a client to group multiple messages and
  acknowledgements into a single unit of work.

In general all these mechanisms apply to all content classes. Certain
content classes may elect to use a simpler model for performance
reasons. This is specifically the case for streamed content, which does
not support persistence, browsing, acknowledgements, or transactions.

Virtual Hosts
-------------

AMQP/Fast has explicit support for virtual hosts. The client chooses a
virtual host as the last stage in negotiating the connection. A virtual
host has its own set of exchanges, queues and access controls.

Exchanges
---------

Exchanges are named objects that act as message filters and multipliers.
Exchanges do not hold messages: rather they copy messages to queues. All
messages are published to exchanges.

Guidelines for implementors:

- Exchanges are not tied to specific content classes.

- A server MUST allow any content class to be sent to any exchange, in any
  mix, depending on the access rights of the producing client.

- A server SHOULD discard messages for which there are no valid queues.
  It MAY log such messages and/or send them to a dead-letter queue.

Queues
------

A queue is a named object that acts as an asynchronous buffer for
messages. Queues may be held in memory, on disk, or some combination.
Each virtual host provides a namespace for queues. Queues hold messages
and distribute them between one or more clients (consumers). A message
published to a queue is never sent to more than one client unless it
is is being resent after a failure or rejection.

Note that mixed content on a queue is generally sent asynchronously
using the appropriate Deliver methods, to those clients that have asked
for it.  The Browse method allows clients to selectively receive single
Basic, JMS or File messages but not Stream messages.

Guidelines for implementors:

- Queues are not tied to specific content classes.

- A server MUST allow any content class to be sent to any queue, in any
  mix, depending on the access rights of the producing client.

Message Persistence
-------------------

A persistent message is held securely on disk and guaranteed to be
delivered even if there is a serious network failure, server crash,
overflow etc.

Messages may be persistent or not, depending on several criteria:

1. In the Basic and JMS content classes, messages are individually
   marked as persistent or non-persistent.
2. In the File content class, messages are always persistent.
3. In the Stream content class, messages are never persistent.

Guidelines for implementors:

- The server MUST do a best-effort to hold persistent messages on a
  fully-reliable storage medium.

- The server MUST NOT discard persistent messages in case of
  overflow but SHOULD use the Channel.Flow method to slow or stop a
  producer when necessary.

- The server MAY overflow non-persistent Basic, JMS or Stream messages
  to persistent storage, or MAY discard non-persistent Basica, JMS or
  Stream messages on a priority basis if the queue exceeds some
  configured maximum size.

Message Priorities
------------------

A high priority message is sent ahead of lower priority messages waiting
in the same queue or subscription.  When messages must be discarded in
order to maintain a specific service quality level the server will first
discard low-priority messages.

Guidelines for implementors:

- A server MUST implement at least 2 priority levels for non-persistent
  messages.  A server MAY implement up to 10 priority levels.

- A server MAY implement all persistent messages as a single priority.

Acknowledgements
---------------

When the client finishes processing a message it tells the server via an
acknowledgement. AMQP/Fast defines different acknowledgement models for
different levels of performance and reliability:

- Acknowledgement by the application after it has processed a message.
- Acknowledgement by the client layer when it has received a message
  and before it passes it to the application.
- Acknowledgement by the server when it has sent a message to the
  client.

The first two models use the same protocol mechanics.  The difference
is when the client API sends the acknowledgement.  The last model uses
a specific protocol option, in the CONSUME method for those classes
that support acknowledgement.

Flow Control
------------

AMQP/Fast flow control is based on these concepts:

1. Allowing the client to specify a prefetch - how many messages
   it will receive without acknowledgement.  We call this "window-based
   flow control".

2. Allowing one peer to halt and restart a flow of messages coming
   from the other peer, using the Channel.Flow method, which we call
   "channel flow control".

Since Publish methods are not acknowledged, there is no window-based
flow control for publishing and the recommended way to pause a publisher
is to use channel flow control.

Guidelines for implementors:

- A server MUST support window-based flow control for clients.

- A server MAY use channel flow control to pause publishers when
  queues or subscriptions overflow.

- A client MUST respect channel flow control when publishing
  messages to the server.

- A server MAY disconnect and/or ban clients that do not respect
  channel flow control methods.

Transactions
------------

AMQP/Fast supports three kinds of transactions, each kind progressively
slower, more complex, and more reliable:

1. Automatic transactions, in which every method is wrapped in a
   transaction. Automatic transactions provide no way to create units
   of work. Every message and acknowledgement is processed as a
   stand-alone transaction.

2. AMQP/Fast transactions, which cover units of work on a per-channel
   basis. Transactions cover messages published, and messages received
   and acknowledged. There is no "start transaction" method: a
   transaction starts when the channel is opened and after every commit
   or rollback. If the channel is closed without a commit, all pending
   work is rolled-back. Nested transactions are not allowed.

3. XA 2-phase transactions, using the XA class.  This is still under
   construction.

The transaction type is chosen on a per-channel basis.

Summary of Classes and Methods
==============================

Goals and Principles
--------------------

We aim to provide a clear and extensible organisation of the classes and
methods that a peer provides. Classes and methods are numbered but also
have constant names (shown in uppercase). The names are chosen to be
consistent across classes, so a similar method in different classes has
the same name and may have similar arguments where appropriate. The full
definition of each class and method is provided later.

We define methods as being either:

- a synchronous request ("syn request"). The sending peer SHOULD wait
  for the specific reply method, but MAY implement this asynchronously.

- a synchronous reply ("syn reply for XYZ").

- an asynchronous request or reply ("async").

A full explanation of classes and methods is provided in the Reference
section of this document.

[see amq_rfc006_quick.txt]

Explanatory Notes
-----------------

Methods are numbered locally per class and there is no attempt to keep a
consistent numbering for similar methods in different classes, nor to
distinguish client or server methods through special numbering schemes.
Method numbering has no significance beyond uniquely identifying the
method.

The grammars use this notation:

- 'S:' indicates data or a method sent from the server to the client.
- 'C:' indicates data or a method sent from the client to the server.
- +term or +(...) expression means '1 or more instances'.
- *term or *(...) expression means 'zero or more instances'.
- Methods are indicated by uppercase names, e.g. OPEN.

Error Handling
==============

Goals and Principles
--------------------

Error handling is a critical aspect of any protocol.  First, we need a
clear statement of what situations can provoke an error.  Secondly, we
need a clear way of reporting errors.  Lastly we need unambiguous error
handling that leaves both peers in a clear state.

The AMQP/Fast error handling model is based on an answer to each of these
questions:

1. Use an exception-based model to define protocol correctness.
2. Use existing standards for error reporting.
3. Use a hand-shaked close to handle errors.

Existing Standards
------------------

The standard for error handling (defined semi-independently in several
IETF RFCs) is the 3-digit reply code. This format has evolved into a
fine-grained tool for communicating success or failure.  It is also
well-structured for expansion as a protocol gets more mature.

The current AMQP reply codes are standard to all protocols in the AMQP
family and are defined in AMQ RFC 011.

The reply code is constructed as follows:

- The first digit - the completion indicator - reports whether the
  request succeeded or not.
- The second digit - the category indicator - provides more information
  on failures.
- The third digit - the instance indicator - distinguishes different
  situations with the same completion/category.

The completion indicator has one of these values:

    1 = ready to be performed, pending some confirmation.
    2 = successful.
    3 = ready to be performed, pending more information.
    4 = failed, but may succeed later.
    5 = failed, requires intervention.

The category indicator has one of these values:

    0 = error in syntax.
    1 = the reply provides general information.
    2 = problem with session or connection.
    3 = problem with security.
    4 = application-specific.

The instance indicator is 0 to 9 as needed to distinguish different
situations.

The Assertion/Exception Model
-----------------------------

AMQP/Fast uses an assertion/exception model that has these goals:

- identify and document all protocol preconditions ("assertions").
- define the exception level caused by any assertion failure.
- define a formal procedure for handling such exceptions.

AMQP/Fast defines two exception levels:

1. Channel exceptions.  These close a single virtual connection.
   A channel exception is raised when a peer cannot complete some
   request because of transient or configuration errors.

2. Connection exceptions.  These close the socket connection.
   A connection exception is raised when a peer detects a syntax
   error, badly-formed frame, or other indicator that the other
   peer is not conformant with AMQP/Fast.

We document the assertions formally in the definition of each class
and method.

Hand-shaked Closure
-------------------

Closing a connection for any reason - normal or exceptional - must be
done carefully.  Abrupt closure is not always detected rapidly, and
in the case of errors, it means that error responses can be lost. The
correct design is to hand-shake all closure so that we close only after
we are sure the other party is aware of the situation.

A peer can close a channel or connection at any time for internal
reasons, or as a reaction to an error.  It sends a Close method to
the other party.  The receiving peer must respond to a Close with a
Close-Ok method.  The closing peer reads methods back until it gets a
Close-Ok, at which point it closes the connection and frees resources.

The Basic Operational Model
===========================

Goals and Principles
--------------------

The basic operational model is designed to provide the simplest
possible message store and forward model.  The basic content methods are
fully synchronous; there is no Consume method, and all message fetching
is done using the Browse method.  Basic messages are not acknowledged,
and cannot be put back onto a queue.

Basic Properties
----------------

These are the properties defined for basic content:

- ContentType (short string)
- ContentEncoding (short string)
- Destination (short string)
- Headers (field table)

The JMS Operational Model
========================

Goals and Principles
--------------------

JMS is a standard API for messaging middleware servers. The AMQP/Fast
JMS implementation is compatible with providers that conform to the SUN
JMS specifications (within the bounds of JMS standardisation, which does
not guarantee interoperability). JMS messages can be published and
consumed using a set of methods that map to the JMS API, including:

- Publishing a message to a "queue".
- Publishing a message to a "topic".
- Creating a consumer for a queue.
- "Subscribing" to a "topic".
- Browsing a destination for messages.

JMS Content Properties
----------------------

These are the properties defined for JMS content:

- ContentType (short string)
- ContentEncoding (short string)
- Destination (short string)
- Headers (field table)
- DeliveryMode (octet)
- Priority (octet)
- CorrelationID (short string)
- ReplyTo (short string)
- Expiration (short string)
- MessageID (short string)
- Timestamp (short string)
- Type (short string)
- UserID (short string)
- AppID (short string)

The File Transfer Operational Model
===================================

Goals and Principles
--------------------

The file transfer operational model is designed for enterprise-wide
file transfer based on the AMQP/Fast queueing and routing models.  For
instance, routing files via the publish and subscribe functionality
of topics and subscriptions.

File content is specifically different from JMS content in that:

- Files are always persistent.
- File transfer is restartable: if a file has been
  partially transferred when a connection is broken, the sender
  can resend just the remainder.

File transfers always happen in two steps:

1. The sender "stages" the file into a temporary area provided by the
   recipient.
2. The sender notifies the recipient of the file, using Publish (a
   client to a server), or Deliver (a server to a client).

A File message is persistent and optimised for restartable transfers
across possibly unreliable network connections.

File Content Properties
-----------------------

These are the properties defined for File content:

- ContentType (short string)
- ContentEncoding (short string)
- Destination (short string)
- Headers (field table)
- Priority (octet)
- ReplyTo (short string)
- MessageID (short string)
- FileName (short string)
- Timestamp (short string)

The Stream Operational Model
============================

Goals and Principles
--------------------

Streaming is intended for multimedia applications: video, music, etc.
A stream consists of an unterminated series of messages, each message
containing a fragment of streamed data.

The AMQP/Fast streaming model separates the compression technology
(codec) from the streaming technology. The streaming model is compatible
with any codec that allows data to be fragmented.

Streamed messages use the full AMQP/Fast queueing and routing mechanisms
with some simplifications:

1. Streamed content is non-persistent though the server MAY hold it on
   disk if needed.

2. Streamed data sent to a queue cannot be shared between multiple
   consumers (each would receive partial streams).

3. Streamed data sent to a subscription via a topic cannot be shared
   between multiple consumers (each would receive partial streams).

A stream producer can use topics to distribute streams to multiple
consumers at once.  Topics can be used - e.g. - to provide different
qualities of the same stream.

A stream consumer can specify the rate at which data is wanted.

Stream Content Properties
-------------------------

These are the properties defined for Stream content:

- ContentType (short string)
- ContentEncoding (short string)
- Destination (short string)
- Headers (field table)
- Priority (octet)
- Timestamp (short string)

Security
========

Goals and Principles
--------------------

We guard against buffer-overflow exploits by using length-specified
buffers in all places.  All externally-provided data can be verified
against maximum allowed lengths whenever any data is read.

Invalid data can be handled unambiguously, by closing the channel or
the connection.

Buffer Overflows
----------------

All data is length-specified so that applications can allocate memory in
advance and avoid deadlocks. Length-specified strings protect against
buffer-overflow attacks.

Denial of Service Attacks
-------------------------

AMQP/Fast handles errors by returning a reply code and then closing the
channel or connection. This avoids ambiguous states after errors.

It should be assumed that exceptional conditions during connection
negotiation stage are due to an hostile attempt to gain access to the
server. The general response to any exceptional condition in the
connection negotiation is to pause that connection (presumably a thread)
for a period of several seconds and then to close the network
connection. This includes syntax errors, over-sized data, or failed
attempts to authenticate. The server implementation should log all such
exceptions and flag or block clients provoking multiple failures.

Reference Section
*****************

[see amq_rfc006_full.txt]

