Meeting Minutes for the LDUP WG Meeting at Oslo, Norway
Minutes taken by John Strassner
Chris Apple - Proposed charter revision status
Posted to the list, no questions on the list. No questions at the WG meeting.
Chris Apple - LDAPv3 replication requirements
The document is in good shape, and we're getting ready for working group Last Call. All that remains is tidying up some loose ends.
Q: if you have sensitive info protected by ACLs, and then delete that tree, what happens on the next access? If the access control is done first, the sensitive info might be temporarily available. So we must ensure that the material protected by the access control must disappear before or at the same time as the access control. So the requirements document must be updated to make this more clear. Note that the problem is on full updates, and we might need a flag in the server to say that the information in the entire tree is temporarily unavailable.
This is a good comment, and should be added to the security considerations session, before we go to Last Call. It also looks like this should be added in the architecture specification and the transfer protocol draft.
Estimated to revise the requirements is within 2-4 weeks, followed by WG Last Call.
John Merrells - LDAPv3 replication architecture
Changes made include the following (please see word attachment): sparse (where only a subset of the entries in a subtree are being replicated) and partial (combination of sparse and fractional) replication deferred - because this was extremely complicated when dealing with update vectors, and querying the list, it was evident that no one needed this right now
Pointed out that by dropping sparse completely, you have less functionality than the base X.500 implementation, since sparse for single-master is different than sparse for multi-master
Read-only replicas were considered, but even this is hard because the information to be replicated is flowing in and out of scope
State-based vs. log-based implementations - this is now TSRs, advisory info added into the document
Consistency model discussion - this is meant to address concerns from IESG. 3 levels of consistency
single server is tight consistency (read and write serializability)
single-master (serializability for writes (single-master) but not for reads) is medium consistency
multi-master where reads and writes are both unserializable is loose consistency
So there is concern that clients will get this wrong. Seems like we need to present more guidance to people.
Paul recommends that any application that updates things that are related to each other (for example) must take explicit mechanisms to deal with this. This is really a paradigm shift for programmers - they must now take explicit measures in their applications. Put another way, LDAP should not be thought of as a single-file database.
LDAP constraints
Info model constraints and operational behavior constraints The purpose of operational behavior constraints are to enforce the data model constraints. Therefore, every time you process an LDAP operation, you must impose these constraints and return an error. Now, in LDUP, you can't actually do this - you have to accept the operations and then fix it in the URP stage. Note also that an LDAP client is used to get the constraint violation immediately. But in a multi-master case this may not be true.
Glue entries for URP.
Same as X.500. Entry has a name, and is used as a placeholder for something that will show up later.
Unique identifier: no prefix byte.
Identifier is a UUID (remove types of identifiers)
Extended ops to protocol draft.
Start and End operations have been moved out into the protocol draft
Replication start has ordering flag
Natural for log-, but not for state-based, system. Problem is that in a state-based system, if you have a huge tree (10M entries) and the line goes down after some period (say, 5M), then you might have to start over.
Replica number table
Traffic will contain CSNs that will contain the replica identifier, which is a lot of data. Hence, we've proposed sending a table that encodes this information.
Replication primitives - these are fully described in the URP doc
Update Resolution Procedures (ed note: it can be a bloodbath ;-):
Distinguished names - can end up with same RDN twice, so we rename and move to Lost+Found. Note that this is a problem if entry is a UID - we need to think more about this.
Orphaned entries - entry loses its parent, we recreate the parent as a glue entry, and glue entry's parent is the Lost+Found container
Distinguished not present - someone has deleted the RDN. We add the value back in and tag it.
Single valued attributes - could end up with multiple values when there is only supposed to be one
Required attributes - required could end up flagged as missing
Extra attributes - attributes that aren't permitted by the class
Duplicate attribute values - must be resolved
Ancestry graph cycle - every entry has to have a parent, but doesn't say that an entry can't be its own parent, or that there can't be a cycle in the DIT. We need a formal statement on these rules
ISSUES Section
Could possibly move sparse replica text into a new informational draft. This met with general approval, but it was recommended that this be done AFTER the existing documents are well on their way to last call
Prescriptive Metadata (John Merrells sent a message earlier on the list; would people please take a look at this and reply)
Replica Identifiers - would be nice to have a simple byte checking operation instead of a more complicated string comparison
Update ordering -
State information representation and new control in order for clients to request state information (e.g., the CSNs). Right now, an ordinary LDAP client can't see the CSNs. Most vendor's clients will, however, have to see this info for admin purposes, so we might consider standardizing it.
Some of the info that we have added for LDUP really should be added as a draft to LDAPEXT
Ordering of updates to meta info - when you replicate, you need to ensure that this occurs at exactly the right time (e.g., all updates that loosen the schema, then updates, then updates that tighten the schema - reverse order for ACLs)
Ed Reed - LDUP replication information model
Ldup-infomod-00.txt was posted, since then an engineering meeting to shrink the schema took place. Shrinking is still continuing ;-)
One request was to not introduce new syntaxes if we could avoid it (because it introduces new matching rules, etc.). Therefore, we dropped ldapAccessPoint in favor of DN to the DSA holding the replica. This will now be an attribute on the replica subEntry class. (DSA entry that has network access of replica that contains this entry)
Drop ldapChangeSequenceNumber syntax and treat it as caseIgnoreString
Update description of CSN to reflect TSRs decision, and to reference the newly added ReplicaID table
Move ldapSubEntry to its own document - proposed to LDAPEXT WG, since it is generally useful
Defer ldapSearchFilter attribute
Still have to define scheduleSubEntry class
Update to follow by end of August
Some concern over storing network address of server in DSA, but this is generally the same problem as that for resolving DNS problems
Major changes from previous draft:
Previous draft had addition and creation CSNs for entries. This was deleted by simplifying the entry re-add semantics (i.e., add entry implies immediately add it immediately preceding removal). This was originally intended to help resolve conflicts for entries that moved in and out of scope, but didn't help. So we are nuking this.
Glue entries created under Lost+Found to take the place of missing superiors for orphaned entries. This help simplifies things by no longer requiring explicit moves to Lost+Found for orphaned entries
Terminology better aligned with the arch draft
CSNs now include the least significant modification number component, a monotonically increasing modification number
Pseudo-code descriptions rearranged to follow a more uniform pattern
Deferred to the next draft:
References to a replication log will be removed when a complete state-based alternative spec is available
appending of Unique Identifier to RDNs, when required to ensure uniqueness, to be made a local server action. This avoids generating explicit rename primitives
Overview of the basic processing structure
For each process in the URP, a fixup procedure can be invoked. Basically, a series of state checks are followed to determine how to process the primitive and whether there have been any constraint violations.
IF a constraint violation is detected, it is fixed and processing then continues. Once all state checks have been completed, the state change requested by the primitive operation may be carried out and propagated to other DSAs.
Attributes can be changed from not distinguished to distinguished, which means that deletions and additions can happen in one place but not in another place. This is a somewhat pathological case.
Delete Not Present has a number of issues. The basic problem is that an attribute deleted at one DSA becomes part of another DSA (i.e., each not having knowledge of the other). This complicates things a lot.
Alison presented two alternatives, and David Chadwick presented a third - detect conflict through rename. The advantage of David's proposal is that it solves the constraint violation, whereas the other two don't). Allison to post to the list.
Access control - may be present at one site but not at another site - so we need to track the access control work that is going on in LDAPEXT. Need to make this more explicit.
Ellen Stokes - Ldup-protocol (not yet published)
Status: no changes in overall approach described at Minneapolis IETF. There are begin and end operations that allow the receiving server to optimize operations (this is where the bulk LDIF proposal is synergistic). Protocol elements are defined. Currently writing descriptions of protocol semantics.
Timeline: will publish as an I-D within a few weeks. ASN.1 defined, beefing up descriptions and ensuring consistency of semantics. Some things we may want to make optional, which may not be applicable if you aren't doing full LDUP replication
Goal of being implemented with current LDAP APIs, but there may be some feedback needed to the API drafts to ensure that all operations can be implemented. May need to add some operations to the API draft. Two areas of concern are error codes and payload encoding.
Extra LDAP Features to Cope with Consistence Levels. Suggestion - Ed's work seems to be correct place to do this to help you understand differences between single- and multi-master profiles. Note that consistency and constraints are two different things. There are mistakes in underlying LDAP specs. For example, it doesn't say that a delete should fail if the entry doesn't exist. Part of this problem is that we like to refer to X.500, and that produces holes. LDUP has discovered these things, but it may be more appropriate to add them into LDAP to do better general good.
Consistency - some prior work in Bayou (session guarantees). Client can say things like it doesn't want to see anything other than a state that it has already read, etc. This would be a follow-on to the update protocols. LDUP addresses by unique identifier, not by DN. Now, if an LDAP client could also address by identifier, this could bring LDAP and LDUP closer together. But this is a change to the core protocol.
Consistency is different in a single- vs. multi-master deployment, but how would the client know what it is talking to? Perhaps we could handle this through explicit administrative configuration. Problem is that old clients will be broken, since they implicitly expect a tighter consistency than what is available in a multi-master system. Perhaps in the old client system, you could have a "preferred" master. Constraints that are needed by applications are not going to be managed by the directory.
What about support for extended LDAP operations? Example is a recursive delete - how would we replicate it? Our response up to now is to ask the people inventing such features how they would replicate it, and up to now, those people have backed down. This is really more of a process - the chairs of LDAPEXT and LDUP, for example, need to ensure that this doesn't happen
Gordon Good/John Merrells/Roger Harrison - Bulk LDIF
Motivation to move this into LDUP is because bulk operations aren't really suited for LDAP. We will have two new protocols, full update and incremental update. Another motivation is synchronization. Tim concurs. Authors to write up the draft formally and post to the list.
