If you are unfamiliar with Python, ArcPy, or script tools, skip to the Getting started with Python, ArcPy, and script tools section below for a list of useful topics.
How project data in your script is found
Whenever you share a result, either as a package or as a service, and the result references a script tool, the script tool is scanned to discover any project data used in the script. When project data is found, it is consolidated into a temporary folder that is either packaged (if you are sharing a package) or uploaded to the server (if you are sharing a service).
When your script is scanned, every quoted string (either single- or double-quotes) used in a Python variable or as an argument to a function is tested to see if it is a path to data that exists. Data, in this case, means
- A layer in the table of contents (ArcMap or ArcGlobe)
- A folder
- A file
- A geodataset, such as a feature class, shapefile, geodatabase, map document (.mxd), and layer file (.lyr)
For the purposes of discussion, only data that is used as input to geoprocessing tools or paths referencing other Python modules is of interest. Output data is also consolidated, but it's not considered to be project data.
Whenever a quoted string is found in the script, the test for data existence proceeds as follows:
- Does the string refer to a layer in the table of contents?
- Does the string contain an absolute path to data (such as "e:\Warehousing\ToolData\SanFrancisco.gdb\streets")?
- Does the string reference data that can be found relative to the script location? The script location is defined as follows:
- The folder containing the script.
- If the script is embedded in the toolbox, the location is the folder containing the toolbox.
- If the script is in a Python toolbox, the location is the folder containing the Python toolbox.
These tests proceed in sequential order. If the test passes, and the data exists, it will be consolidated, with one exception: if you are sharing a service, the server's data store is examined to determine if the data resides in the data store. If it resides in the data store, then it is not consolidated.
Examples
The examples below are based on this folder structure:
Relative paths to data sets
The following technique of finding data relative to the location of the script is a common pattern, especially for services created for ArcGIS 10.0. The ToolData folder contains SanFrancisco.gdb. Within SanFrancisco.gdb is a feature class named Streets. In the code sample below, the path to the ToolData folder and the datasets inside are constructed relative to the location of the script (the same directory as the Warehouse.tbx).
import arcpy
import os
import sys
# Get the directory the script lives in.
# Folders and data will be found relative to this location.
#
scriptPath = sys.path[0]
# Construct paths to ../ToolData/SanFrancisco.gdb/Streets and
# ../ToolData/Warehouse.lyr
streetFeatures = os.path.join(scriptPath, "ToolData\\SanFrancisco.gdb\\Streets")
streetLyr = os.path.join(scriptPath, "ToolData\\Warehouse.lyr")
In the code above, the variables streetFeatures and streetLyr will be tested to see if they reference data that exists. These datasets will be consolidated and uploaded to the server (unless the folder they reside in has been referenced as part of the server's data store).
Relative paths to folders
The ToolData folder itself can be referenced and used as a starting point to reference items. In the code sample below, the path to the ToolData folder is constructed relative to the location of the script (the same directory as the Warehouse.tbx).
import arcpy
import os
import sys
# Get the directory the script lives in.
# Folders and data will be found relative to this location.
#
scriptPath = sys.path[0]
# Construct paths to the ToolData folder
toolDataFolder = os.path.join(scriptPath, "ToolData")
# Construct path to items inside the folder
streetFeatures = os.path.join(toolDataFolder, "SanFrancisco.gdb\\Streets")
In the code above, the variable toolDataFolder becomes a relative path to a folder of various items that can be referenced throughout your Python script. This ToolData folder will be consolidated—all its contents (with the exception of subfolders as noted above) will be packaged or uploaded to the server (unless the ToolData folder is part of the server's data store).
Note that folder contents are copied, not individual files when referencing a folder. For example, in the above code, despite the fact no explicit path to the Warehouse.lyr has been created, the file will be consolidated as it exists in the referenced folder.
Absolute path to a geodataset
An absolute path is one that begins with a drive letter, such as e:/, as shown in the code sample below.
import arcpy
import os
streetFeatures = 'e:/Warehousing/ToolData/SanFrancisco.gdb/Streets'
In the above code, the Streets dataset, and all other data it depends upon (such as relationship classes and domains), will be consolidated.
Hybrid example
import arcpy
import os
toolDataPath = r'e:\Warehousing\ToolData'
warehouseLyr = os.path.join(toolDataPath, "Warehouse.lyr")
In the above code, the entire contents of the ToolData folder is consolidated. Since the folder contents (minus subfolders) is consolidated, Warehouse.lyr will be consolidated as well, along with any data referenced by Warehouse.lyr.
Referencing layers as project data
A not as common workflow of using layers as project data can result in significant performance improvements for your Python script tool. The above Python snippets use full paths to feature classes and layer files. When a geoprocessing service is executed, it must first open the dataset, and opening a dataset is a performance hit. Using layers in your script keep the data opened and cached for faster execution. The following image shows how layers within the ArcMap Table of Contents are matched up and used within the Python script.
The two layers from the Table of Contents are used inside the script tool. The variables point to simple strings which match the layer names inside the map document. When publishing this workflow to ArcGIS Server, the data will be consolidated and moved to the server (if not referenced in the data store) and the service will hold a reference to the layers in memory. Within that service the tool will find and use layers through this name match.
Forward versus backward slashes
The Windows convention is to use a backward slash (\) as the separator in a path. UNIX systems use a forward slash (/).
Backward slash in scripting
Programming languages that have their roots in UNIX and the C programming language, such as Python, treat the backslash (\) as the escape character. For example, \t signifies a tab. Since paths can contain backslashes, you need to prevent backslashes from being used as the escape character. The easiest way is to convert paths into Python raw strings using the r directive, as shown below. This instructs Python to ignore backslashes.
thePath = r"E:\data\telluride\newdata.gdb\slopes"
Importing other Python modules
Your script may import other scripts that you developed. For example, the code below shows importing a Python module named myutils.py, which is found in the same directory as the parent script and contains a routine named getFIDName.
import arcpy
import myutils
inFeatures = arcpy.GetParameterAsText(0)
inFID = myutils.getFIDName(inFeatures)
Whenever an import statement is encountered, the following order is used to locate the script:
- The same folder as the script. If the script is embedded in the toolbox, the folder containing the toolbox is used.
- The folder referenced by the system's PYTHONPATH variable.
- Any folder referenced by the system's PATH variable.
Another technique for referencing modules to import is to use the sys.path.append method. This allows you to set a path to a folder containing scripts that you need to import.
import arcpy
import sys
import os
# Append the path to the utility modules to the system path
# for the duration of this script.
#
myPythonModules = r'e:\Warehousing\Scripts'
sys.path.append(myPythonModules)
import myutils # a Python file within myPythonModules
In the above code, note that the sys.path.append method requires a folder as an argument. Since r'e:\Warehousing\Scripts' is a folder, the entire contents of the folder will be consolidated. The rules for copying folder contents apply here as well—everything in the folder is copied except for subfolders that are not geodatasets.
Tool validation code
If you have experience writing script tools, you may be providing your own tool validation logic. Validation logic is implemented with Python, and your validation code will be scanned for project data and modules, just like any other Python script. For example, your validation logic may open a folder (for example, d:\approved_projections) containing projection files (.prj) to build a choice list of spatial references users can choose when they execute your tool. This folder is not a tool parameter; it's project data used within your tool validation script. The same rules described above for Python scripts apply here, and the consequence is that the d:\approved_projections folder will be consolidated and copied to the server (unless it's found in the server's data store).
Third-party libraries
Third-party modules (any module that is not part of the core Python installation) are not consolidated. You need to ensure the module is installed on the machine where the packaged is unpacked. You should provide documentation for your tool and package that specifies what third-party modules are required. This does not apply to the numpy or matplotlib modules which are installed with ArcGIS Desktop.
Getting started with Python, ArcPy, and script tools
If you are unfamiliar with Python, ArcPy, and script tools, the table below lists a few topics that will help you get started.
Help topic | Contents |
---|---|
Basic concepts of creating your own geoprocessing tools. | |
Introductory topics to Python and ArcPy. These topics lead you to more detailed topics about Python and the ArcPy site package. | |
Introductory topic on creating custom script tools using Python. | |
Once you've become familiar with the process of creating a script tool, this topic is referred to often as it explains in detail how to define script tool parameters. |