1 The OpenPegasus Binary Protocol FAQ
2 ===================================
4 This FAQ (Frequently Asked Questions) hopefully addresses questions you might
5 have about the the OpenPegasus Binary Protocol. If you find your question was
6 not addressed, please ask your question on the OpenPegasus mailing list and
7 request that the answer be included in this document.
9 What is the Binary Protocol?
10 ============================
12 The binary protocol is a fast protocol for client-server communication. It
13 allows local clients to send binary messages to the OpenPegasus server and
14 receive binary responses. This protocol is much faster than the default
15 XML protocol. When the binary protocol is enabled, local clients use it to
16 communicate with the OpenPegasus server. Examples of local clients include:
17 (1) out-of-process providers making up-calls, (2) any provider making a local
18 connection with the CIMClient class, (2) any local process making a local
19 connection with the CIMClient class.
21 Why use the Binary Protocol?
22 ============================
24 The main reason to use the binary protocol is to improve performance of the
25 server and clients. Some operations execute as much as 4 times faster with
28 How big are binary messages?
29 ============================
31 Binary messages are slightly larger than XML messages for two reasons:
32 (1) Strings are transmitted using 2-byte characters and (2) Objects are
33 aligned on 8-byte boundaries.
35 What are the basic rules for encoding messages?
36 ===============================================
38 The encoding rules were designed for speed rather than size. The basic rules
41 (1) All basic types are aligned on 8-byte boundaries. For example, a uint32
42 starts on an 8-byte boundary and is followed by 4 padding bytes so that
43 the next type is aligned on an 8-byte boundary. This alignment has 2
44 advantages: (1) it allows any basic type to be dereferenced directly in
45 the buffer without having to relocate it and (2) it is inexpensive to
46 calculate the alignment of the next type.
48 There are 2 other data alignment techniques we could have chosen:
50 (*) Don't align at all. Just pack types into the buffer end-to-end.
51 This technique yields smaller messages but sacrifices
52 performance since types cannot be assigned directly to or
53 from the data buffer (some operating systems generate data
56 (*) Align types on their "natural boundaries". This means that
57 a type should be aligned on boundaries divisible by its size.
58 For example, a 2-byte integer should be aligned on a 2-byte
59 boundary, or a 4-byte integer should be aligned on a 4-byte
60 boundary. This alignment technique allows types to be directly
61 assigned to or from the data buffer. However, aligning the data
62 buffer for the next type is slightly more expensive since it
63 requires a few extra instructions to compute the alignment
64 boundary than our approach.
66 (2) Types are serialized into the message in their native representations.
67 We do not change the representation to either big endian or little
68 endian. Instead, the recipient of the message is responsible for
69 converting the data into its native representation. If both processes
70 have the same native representation, then the reordering of bytes can
71 be avoided (this is always the case with local processes). This policy
72 has been referred to as "reader makes right". That is, the reader is
73 responsible for making the incoming data into the "right"
76 (3) All arrays are represented by their size followed by their elements.
77 The size is always 4 bytes with 4 extra bytes of padding. In this way,
78 the elements always begin on an 8-byte boundary. The elements are packed
79 end to end with no padding.
81 (4) Strings are represented like arrays (size plus elements).
83 (5) Boolean are represented as a single byte, either 0 or 1.
85 (6) Complex objects that have optional elements or boolean elements, often
86 employ a single 4-byte bit mask that indicates which flags are true
87 and which elements are present in the network buffer. For example,
88 the CIMProperty representation has a bit mask that indicates whether
94 (*) has a non-empty references class .
95 (*) has a non-empty class origin.
97 This save considerable space. For example, if there are no qualifiers,
98 then we save 8 bytes that would be needed to represent an empty
101 What is the layout of a binary message?
102 =======================================
104 Binary messages are comprised of a header followed by a body. The header has
105 the following elements:
107 (1) Magic number - contains 0xF00DFACE.
108 (2) Version number - 1 for the first version.
109 (3) Flags - flags used to represent boolean options of the message.
110 (4) Message ID - same as the message ID in a CIM message.
111 (5) Operation - an integer representing the CIM operation, given as follows:
116 (*) IndicationDelivery = 4 (binary version not implemented)
118 (*) DeleteInstance = 6
120 (*) CreateInstance = 8
122 (*) ModifyInstance = 10
123 (*) EnumerateClasses = 11
124 (*) EnumerateClassNames = 12
125 (*) EnumerateInstances = 13
126 (*) EnumerateInstanceNames = 14
129 (*) AssociatorNames = 17
131 (*) ReferenceNames = 19
134 (*) GetQualifier = 22
135 (*) SetQualifier = 23
136 (*) DeleteQualifier = 24
137 (*) EnumerateQualifiers = 25
138 (*) InvokeMethod = 26
140 Does the binary protocol use HTTP?
141 ==================================
143 Yes. The binary protocol uses the existing OpenPegasus HTTP infrastructure.
144 It preserve the same headers as the conventional protocol.
146 Does the binary protocol define new HTTP headers?
147 =================================================
149 Yes. It defines two new headers:
151 Content-Type: application/x-openpegasus
152 Accept: application/x-openpegasus
154 The first header is borne by both binary requests and binary responses.
155 It indicates that the content (payload) contains an OpenPegasus binary messages.
157 The second header is sent by a request and indicates that the client can
158 handle OpenPegasus binary responses.
160 The client can combine these headers to achieve 4 different behaviors:
162 (1) Binary request/Binary response:
164 Content-Type: application/x-openpegasus
165 Accept: application/x-openpegasus
167 (2) Binary request/XML response:
169 Content-Type: application/x-openpegasus
171 (3) XML request/binary response:
173 Accept: application/x-openpegasus
175 (4) XML request/XML response:
179 Only 1 and 4 can be achieved without minor code changes to OpenPegasus.
181 How does protocol versioning work?
182 ==================================
184 The binary messages carries a version number in the header. This will be used
185 to support backwards compatibility with clients. The server must never be
186 modified to send version N+1 messages to version N clients.
188 Does the binary protocol support remote communication?
189 ======================================================
191 Yes, although there is no official SDK interface for enabling it. To enable
192 it, one must obtain the CIMClientRep from the CIMClient instance and set
193 the following data members to true.
195 CIMClientRep::_binaryRequest
196 CIMClientRep::_binaryResponse
198 The following code fragment shows how one might do this in a program.
200 static void _SetBinaryRequest(CIMClient& client, Boolean flag)
202 CIMClientRep* rep = *(reinterpret_cast<CIMClientRep**>(&client));
203 rep->setBinaryRequest(flag);
206 static void _SetBinaryResponse(CIMClient& client, Boolean flag)
208 CIMClientRep* rep = *(reinterpret_cast<CIMClientRep**>(&client));
209 rep->setBinaryResponse(flag);
215 _SetBinaryRequest(client, true);
216 _SetBinaryResponse(client, true);
218 client.connect("localhost", 22000, String(), String());
220 This forces a remote binary connection.
222 What is on-demand de-serialization?
223 ===================================
225 The binary protocol supports a feature we call "on-demand de-serialization".
226 When using out-of-process providers, data may be de-serialized unnecessarily.
227 Consider the following sequence of events.
229 (1) The client sends an EnumerateInstances request to the server.
230 (2) The server de-serializes the request.
231 (3) The server serializes the request for the provider agent.
232 (4) The provider agent de-serializes the request.
233 (5) The provider agent obtains response.
234 (6) The provider agent serializes the response for the server.
235 (7) The server de-serializes the request.
236 (8) The server serializes the request for the client.
237 (9) The client de-serializes the request.
239 The on-demand de-serialization feature eliminates the de-serialization of the
240 returned instances, saving them in a data buffer. Then in step 8, the data
241 buffer is sent to the client. This optimization avoids one de-serialization and
242 one serialization. For the EnumerateInstances operation, this optimization
243 alone doubles the speed of servicing this operation.
245 On-demand de-serialization is implemented for the following operations:
247 (1) EnumerateInstances
252 Where is the binary encoding/decoding code located?
253 ===================================================
255 The source code for encoding and decoding binary requests and responses is
256 all included in a single module. The source files are located here:
258 pegasus/src/Pegaus/Common/BinaryCodec.h
259 pegasus/src/Pegaus/Common/BinaryCodec.cpp
261 The source code that implements the encoding of objects themselves is located
264 pegasus/src/Pegaus/Common/CIMBuffer.h
265 pegasus/src/Pegaus/Common/CIMBuffer.cpp
267 How do you build OpenPegasus with binary protocol support?
268 ==========================================================
270 To build OpenPegasus with binary protocol support, define the following
271 environment variable first:
273 $ export PEGASUS_ENABLE_PROTOCOL_BINARY=true
275 Are there further improvements that could be made to the protocol?
276 ==================================================================
280 (1) Excessive copying is required to convert CIMBuffer objects into
281 Buffer objects. CIMBuffer could be reimplemented to use Buffer
282 as its representation. Then it would be possible to "swap" their
283 implementations rather than copying one to the other.
285 (2) It might have been better to align objects on their natural boundaries
286 rather than on 8-byte boundaries. This would probably reduce the
287 message size by 25% or so.
289 (3) The on-demand de-serialization should probably be extended to operations
290 other than just GetInstance and EnumerateInstances. For example, the
291 following operations would benefit the most.
293 (*) EnumerateInstanceNames
299 (4) The binary protocol should be extended to support the pull-operations
300 whenever they are implemented for XML.