Tolerances, expressed in the coverage's units, are used for coverage automation and update steps, such as coverage registration, feature snapping, and coordinate spacing. These tolerances affect the coverage resolution, since they specify the amount of coordinate movement allowed during an operation. The greater the movement allowed, the lower the resulting resolution. For example, when you digitize or edit with node snapping turned on, two nodes are automatically snapped together if they fall within a given distance of each other; that distance is the node snap tolerance. If the feature coordinates are in digitizer units, the tolerances are likely in inches; if the features are projected, the tolerances are likely in feet or meters. The tolerances you set control editing behavior in the ArcInfo Workstation ArcEdit application.
The Coverage Properties dialog box lists the current topology for Fuzzy and Dangle tolerance values. Next to each value, the words verified or unverified, indicate if a tolerance was explicitly set during the editing process. If you want to change the default tolerances to values that are more appropriate for the type and scale of features in the coverage; when you do so on the Tolerances tab, Default disappears. You can set tolerances using the Tolerance tool; however, you need an Desktop Advanced license to use this geoprocessing tool. When you open the Coverage Properties dialog box after running the tool or command, Verified appears next to the tolerance that was used.
The following lists the 8 tolerances you should understand when cleaning or editing coverages:
- Arc snap tolerance: Distance a newly added arc will be extended to close an undershoot.
- Dangle: Dangling arcs shorter than the dangle length are deleted during the Clean operation.
- Fuzzy tolerance: The maximum distance coordinates can shift during Clean, Buffer, and overlay operations. It is the distance used to remove arc vertices and detect intersections.
- Grain tolerance: The controlled distance between vertices when adding splines and curves (for example, culs-de-sac).
- Node snap tolerance: Distance within which a new node is snapped to an existing node.
- Root mean square (RMS) error: A measure of tic registration accuracy during digitizing and coverage transformation.
- Tic match tolerance: The maximum registration error allowed when registering a map on the digitizer. This is used to help ensure accurate map registration (that is, to keep RMS error low).
- Weed tolerance: The minimum distance between points in an arc used during digitizing and to generalize existing arcs.
Note, you can only edit a coverage using ArcInfo Workstation.
Processing tolerances of coverages
Arc snap tolerance
The arc snap tolerance is the distance a new arc will be extended to intersect with an existing arc. A node is created at the new intersection of the connecting arcs.
The recommended value for an arc snap tolerance is 0.05 inches (0.127 centimeters) or equivalent coverage units for the given input map scale.
Dangle length
A dangling arc has the same polygon on its left and right sides (as defined by the polygon internal number) and at least one dangling node. The dangle length defines the minimum allowed length for a dangling arc in a coverage. The Clean tool deletes dangling arcs less than the dangle length.
Recommended values for dangle length often vary with the type of map features being automated. Typically, a dangle length of 0 is recommended when the coverage will contain linear features. For polygons, a dangle length of 0.05 inches (0.127 centimeters) or equivalent coverage units is often recommended. Negative values are not accepted.
Fuzzy tolerance
Coverage resolution is influenced by the fuzzy tolerance, which represents the minimum distance separating all arc coordinates (nodes and vertices) in a coverage. By definition, it also defines the distance a coordinate can move during certain operations. The fuzzy tolerance is an extremely small distance used to resolve inexact intersection locations due to limited arithmetic precision of computers. Fuzzy tolerance values typically range from 1/10,000 to 1/1,000,000 times the width of the coverage extent defined in the coverage bnd.adf file.
When a coverage is processed using certain tools (such as Clean), coordinates within the fuzzy tolerance of each other are considered equal (that is, at the same location). Thus, some coordinates are moved to the location of other coordinates. To keep this movement small, keep the fuzzy tolerance small (that is, smaller than 0.002 inches). Using double-precision coordinates for coverages allows you to define extremely small fuzzy tolerances. The fuzzy tolerance unit of measure is the same as that of the input coverages.
Because arc coordinates can move by as much as the fuzzy tolerance distance, many potential problems in a coverage can be resolved by processing a coverage with tools that use the fuzzy tolerance. These include handling extremely small overshoots or undershoots, automatic sliver removal of duplicate segments, and coordinate thinning along arcs.
The fuzzy tolerance is used by the Clean tool. This is the distance within which the Clean tool is allowed to move features, eliminate duplicate nodes, create nodes at intersections of lines, and eliminate duplicate features. The fuzzy tolerance is measured in coverage units. When using the Clean tool, it is critical that an appropriate fuzzy tolerance be assigned so that necessary features are not eliminated by mistake. A fuzzy tolerance that is too small may cause two kinds of problems: arc intersections may not be created or extremely small slivers may not be successfully cleaned. Conversely, a fuzzy tolerance that is too large may cause some features to move too much (for example, surveyed section corners within a Township Range grid), polygons to collapse, or arcs to merge that should not.
The fuzzy tolerance is specified as a variable in many tools (for example, Clean, Clip, Buffer, Erase, Intersect, Identity, Split, and Union). You can specify a value or use the default value in the coverage tolerance file. ArcGIS Desktop calculates the default fuzzy tolerance in the following order:
- If a fuzzy tolerance is not specified, it is read from the tolerance (TOL) file. If there is no tolerance file or no value for fuzzy tolerance within the file, an error message is produced and processing stops.
- If the coverage range in the bnd.adf file is not specified, the fuzzy tolerance is set to 0.001, even though a fuzzy tolerance value was specified on the command line or in the tolerance file.
- If the tool performs a line segment intersection operation (such as for Clean) and the coverage coordinates are single precision, then the coverage boundary range and maximum range values are used to calculate the minimum required fuzzy tolerance. If this tolerance is greater than the tolerance resulting from the two steps above, the greater value is used and the tolerance file is updated with the new value. Otherwise, the fuzzy tolerance is not changed.
The fuzzy tolerance for a digitizer precision of 0.002 inches is calculated as follows:
(scale denominator/number of inches per coverage unit) * 0.002
For example, 1:250,000 in coverage units of feet yields the following:
(250,000/12) * 0.002 = 41.660
Small fuzzy tolerance values (such as 1/100,000 times the extent of the bnd.adf) are recommended. If the fuzzy tolerance you specify is too small, a minimum tolerance is automatically calculated. This occurs most frequently for single-precision coverages.
Common fuzzy tolerance values
| Input scale for digitizing | Coverage units | Fuzzy tolerance (on the ground) |
|---|---|---|
1:250,000 | Feet | 41.660 |
Meters | 12.700 | |
1:100,000 | Feet | 16.620 |
Meters | 5.080 | |
1:63,360 | Feet | 10.560 |
1:62,500 | Meters | 3.210 |
1:24,000 | Feet | 4.000 |
Meters | 1.219 | |
1:6,000 | Feet | 1.000 |
Meters | 0.304 |
Grain tolerance
The grain tolerance controls the number of vertices in an arc and the distance between them along curved lines. The smaller the grain tolerance, the closer vertices can be. The grain tolerance is also used for densifying the number of arcs in a curve. Whereas the grain tolerance will affect the shape of newly created curves, it has no effect on shape when used to densify existing arcs.
The recommended grain tolerance, 0.02 inches (0.0508 centimeters) or equivalent coverage units, is the same value recommended for the weed tolerance. When creating new arcs, the weed tolerance controls the distance between vertices along straight arcs, and the grain tolerance controls the distance along curved arcs.
Node snap tolerance
The node snap tolerance is the minimum distance within which two nodes will be joined (matched) to form one node.
The matching occurs as arcs are added or edited and nodes are moved or added. For example, as the arc represented by the dashed line in the figure above is added, its nodes will automatically snap to any existing nodes within the node snap tolerance.
The recommended value for node snapping is 0.05 inches (0.127 cm) or equivalent coverage units. This is a very liberal node snap tolerance for accurate coverage automation.
RMS error
The root mean square value represents the amount of error between original and new coordinate locations calculated by the transformation process. ArcGIS Desktop Advanced automatically calculates the root mean square error (or tic registration error) when tics are used to register a map on the digitizer and during transform (projection) operations. The lower the RMS error, the more accurate the digitizing or transformation is.
To register a map, tic locations are digitized and compared to the stored x,y locations for coverage tics. This process defines a coordinate transformation, that is, a transformation scale, a rotation, and a translation (shift) in the x and y directions. This transformation is then applied to all coordinates entered at the digitizer, rendering them as coverage units.
RMS is reported in both digitizer units and coverage units. To maintain highly accurate geographic data, the RMS should be kept under 0.004 inches (or its equivalent measurement in the coordinate system being used). For less accurate data, the value can be as high as 0.008 inches or its equivalent measure. The tic match tolerance (described below) can be used to ensure a low RMS error.
An RMS error is also calculated by the Transform tool. When using the Transform tool, RMS represents the accuracy at which old coverage coordinates will be transformed into the new coverage. It is computed by comparing tic locations in the old coverage to input tics in the new coverage. You can learn more about transformations and RMS errors in the Transform tool help files.
Tic match tolerance
The tic match tolerance, reported in coverage units, is used to ensure accurate map registration on a digitizer. It is the maximum allowed distance between an existing tic and a tic being digitized. Beyond this, the digitizing error is unacceptable and requires that the map be registered again. The tic match tolerance is used to ensure a low RMS error during map registration on a digitizer. It measures how accurately each tic location in the coverage matches the digitized location. A tic registration error is calculated automatically for each tic when the map is registered on the digitizer. You are asked to reregister any tic location that is farther than the match tolerance from the tic.
In the above example, tic 3 must be registered again because it is farther than the tic match tolerance from the expected tic location.
A recommended tic match tolerance will vary depending on the quality of the data being automated. The value should be no higher than 0.004 inches (0.01016 centimeters) for highly accurate map data and 0.008 inches (0.02032 centimeters) for maps requiring less accuracy.
You set the tic match tolerance with the Tolerance tool. Once set, the tolerance is used whenever the tics are redigitized in ArcGIS Desktop.
Weed tolerance
The weed tolerance is the minimum allowable distance between any two vertices along an arc. The weed tolerance is used to reduce the number of coordinates in an arc and is a parameter that can be set before adding arc features or to generalize existing arcs.
When adding a new arc, a new vertex within the weed tolerance of the previous vertex is disregarded.
When weeding existing arcs, the Douglas-Peucker algorithm is used to weed coordinates within each arc. Trend lines are created from each arc’s starting point to other vertices in the arc, and the weed tolerance is used to identify vertices to be weeded that fall within the tolerance of each trend line.
A weed tolerance value of 0.02 inches (0.0508 centimeters) or equivalent coverage units is recommended.
The coverage TOL file
A coverage's tolerance (tol.adf) file contains values for the coverage's fuzzy tolerance, dangle length, and tic match tolerance. Since these tolerances affect coverage processing and help define the resolution of a coverage, it is important to keep track of their values. Tolerance values can be set for many operations in ArcGIS Desktop. For example, both the fuzzy tolerance and dangle length are specified as inputs for coverage tools in which a geographic analysis is performed and topology is created (such as Clean, Buffer, and all Overlay tools).
A tolerance set with the Tolerance tool does not immediately reflect the actual resolution of the coverage. The coverage must be processed using the new tolerance before it will accurately reflect the coverage resolution. Therefore, each tolerance has a verified indicator. Only if a coverage has been processed with the specified tolerance is that tolerance a verified one. The verification status of each tolerance is shown in the tolerance listing.
Another way to view the tolerances of a coverage takes place in ArcCatalog. There, you can access the general tab of the Coverage Properties dialog box to see the coverage's fuzzy and dangle topology tolerances.
Calculating a tolerance for a given map scale
The following formulas can be used to calculate measures in feet or meters for various input map scales.
If the coverage is stored in feet
tol(feet) = tol(inches) * scale / 12If the coverage is stored in meters
tol(meters) = tol(centimeters) * scale / 100Note that scale in these formulas is the denominator of the representative fraction (for example, the value 250,000 from a map whose scale is 1:250,000).
For example, 0.002 inches at a map scale of 1:63,360 is equivalent to 10.56 feet or 3.218 meters.