The custom tool you are attempting to publish is based on a DLL. DLLs are not published to the server, as custom tools typically need to be installed and possibly configured.
Solution
Ensure the custom tool has been installed and works on the ArcGIS Server machine before publishing your geoprocessing service.
More information
See Deploying custom .NET and C++ toolsor Deploying custom Java function toolstopics for more information about deploying custom tools to ArcGIS Server.